[Question] dfget fails to pull images #252

BlueBlue-Lee · 2018-11-30T08:16:57Z

Question

After installed all component of dragonfly and finished all configuration according to this document, I failed to pull images via dragonfly with great possibility.

when failed, check the dfclient.log:

[2018-11-30 16:14:54,289] INFO sign:8797-1543565691.582 lineno:64 : do register to 127.0.0.1,remainder:['127.0.0.1']
[2018-11-30 16:14:54,771] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:2,cdn status:RUNNING,cdn sucCount:2', u'code': 602} and sleep 1.142 ...
[2018-11-30 16:14:54,916] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.890 ...
[2018-11-30 16:14:55,921] INFO sign:8626-1543565671.974 lineno:60 : local http result:20971520 for path:/rate/ and cost:0.001
[2018-11-30 16:14:55,957] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.064 ...
[2018-11-30 16:14:56,813] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.013 ...
[2018-11-30 16:14:57,026] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.577 ...
[2018-11-30 16:14:57,832] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.743 ...
[2018-11-30 16:14:58,610] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.613 ...
[2018-11-30 16:14:59,295] ERROR sign:8797-1543565691.582 lineno:81 : register to node:127.0.0.1 error
Traceback (most recent call last):
  File "/usr/local/bin/component/httputil.py", line 67, in register
    schema, node), data=params, timeout=(2.0, 5.0))
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 555, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8002): Read timed out. (read timeout=5.0)
[2018-11-30 16:14:59,300] INFO sign:8797-1543565691.582 lineno:64 : do register to 127.0.0.1,remainder:[]
[2018-11-30 16:14:59,582] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.895 ...
[2018-11-30 16:15:00,084] ERROR sign:8797-1543565691.582 lineno:136 : init fail but try back down
Traceback (most recent call last):
  File "/usr/local/bin/dfget", line 128, in <module>
    register_result = env.init()
  File "/usr/local/bin/env.py", line 148, in init
    paramparser.cmdparam.identifier)
  File "/usr/local/bin/env.py", line 84, in register
    nodes, paramparser.cmdparam.url, task_url, port, http_path, md5, identifier)
  File "/usr/local/bin/component/httputil.py", line 86, in register
    raise Exception("register result:%s" % result)
Exception: register result:{u'code': 607}
[2018-11-30 16:15:00,085] INFO sign:8797-1543565691.582 lineno:599 : download fail and not back source
[2018-11-30 16:15:00,087] INFO sign:8797-1543565691.582 lineno:94 : |UNKNOWN|http://hub.c.163.com/v2/library/php/blobs/sha256:2871138c7173567814c6bda10b07b3d772bfafb53ffefe66a4e44e530f4127bc|-1|0|UNKNOWN|com_ops_dragonfly|8.505|
[2018-11-30 16:15:00,087] INFO sign:8797-1543565691.582 lineno:111 : download FAIL cost:8.505s length:-1 reason:1001
[2018-11-30 16:15:00,234] INFO sign:8626-1543565671.974 lineno:60 : local http result:20971520 for path:/rate/ and cost:0.002
[2018-11-30 16:15:00,271] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:4,cdn status:RUNNING,cdn sucCount:4', u'code': 602} and sleep 0.956 ...
[2018-11-30 16:15:01,240] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:4,cdn status:RUNNING,cdn sucCount:4', u'code': 602} and sleep 1.511 ...
[2018-11-30 16:15:01,484] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.554 ...

The text was updated successfully, but these errors were encountered:

BlueBlue-Lee · 2018-11-30T08:20:05Z

# python --version
Python 2.7.13

# docker ps 
CONTAINER ID        IMAGE                                                            COMMAND                  CREATED             STATUS              PORTS                              NAMES
789ccbf39533        registry.cn-hangzhou.aliyuncs.com/alidragonfly/supernode:0.2.0   "/root/start.sh"         About an hour ago   Up About an hour    0.0.0.0:8001-8002->8001-8002/tcp   inspiring_johnson

# ps aux | grep dfdaemon
root     51394  0.0  0.2 316180 10484 pts/0    Sl   15:22   0:00 dfdaemon --registry hub.c.163.com --verbose

# cat /etc/dragonfly.conf 
[node]
address=127.0.0.1

BlueBlue-Lee · 2018-11-30T08:22:40Z

@lowzj @allencloud Any ideas? Thanks a lot.

BlueBlue-Lee · 2018-11-30T08:31:09Z

Also fail to download file with dfget

# dfget -u 'https://github.com/dragonflyoss/Dragonfly/blob/master/do/images/logo.png' -o /tmp/logo.png
--2018-11-30 16:29:38--  https://github.com/dragonflyoss/Dragonfly/blob/master/do/images/logo.png
current user[root] output path[/tmp/logo.png]
dfget version:0.2.0
workspace:/root/.small-dragonfly/ sign:16021-1543566578.479
client:127.0.0.1 connected to node:127.0.0.1
init success...
start download by dragonfly
migrated to node:127.0.0.1
download from source
download SUCCESS(0) cost(6.978s) length:129202 reason:3

dfclient.log:

[2018-11-30 16:29:38,483] INFO sign:16021-1543566578.479 lineno:51 : cmd params:Namespace(callsystem=None, console=False, dfdaemon=False, filter=None, header=None, identifier=None, locallimit=None, md5=None, node=None, notbs=False, output='/tmp/logo.png', pattern=None, showbar=False, timeout=None, totallimit=None, url='https://github.com/dragonflyoss/Dragonfly/blob/master/do/images/logo.png', version=False),python version:sys.version_info(major=2, minor=7, micro=13, releaselevel='final', serial=0)
[2018-11-30 16:29:38,484] INFO sign:16021-1543566578.479 lineno:107 : target file path:/tmp/logo.png
[2018-11-30 16:29:38,485] INFO sign:16021-1543566578.479 lineno:133 : sysLang:UTF-8
[2018-11-30 16:29:38,485] INFO sign:16021-1543566578.479 lineno:137 : taskFileName:logo.png-16021-1543566578.479 and cid:127.0.0.1-16021-1543566578.479
[2018-11-30 16:29:38,485] INFO sign:16021-1543566578.479 lineno:140 : task url:https://github.com/dragonflyoss/Dragonfly/blob/master/do/images/logo.png
[2018-11-30 16:29:38,486] ERROR sign:16021-1543566578.479 lineno:58 : request local http path:/check/ error
Traceback (most recent call last):
  File "/usr/local/bin/component/netutil.py", line 54, in request_local
    res = urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>
[2018-11-30 16:29:38,487] INFO sign:16021-1543566578.479 lineno:60 : local http result: for path:/check/ and cost:0.001
[2018-11-30 16:29:38,496] INFO sign:16021-1543566578.479 lineno:60 : local http result:[email protected] for path:/check/ and cost:0.004
[2018-11-30 16:29:38,496] INFO sign:16021-1543566578.479 lineno:64 : do register to 127.0.0.1,remainder:['127.0.0.1']
[2018-11-30 16:29:40,927] INFO sign:16021-1543566578.479 lineno:93 : do register result:{u'code': 200, u'data': {u'pieceSize': 4194304, u'fileLength': -1, u'taskId': u'e602b9db61726ed79e4156a932fc0fa66d9a5958082736af3b1f94f0779e6f31'}} and cost 2.431
[2018-11-30 16:29:40,933] INFO sign:16021-1543566578.479 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 0.714 ...
[2018-11-30 16:29:41,652] INFO sign:16021-1543566578.479 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.287 ...
[2018-11-30 16:29:42,945] INFO sign:16021-1543566578.479 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.554 ...
[2018-11-30 16:29:44,505] ERROR sign:16021-1543566578.479 lineno:119 : pull piece task fail:{u'msg': u'cdn status is fail', u'code': 604} and will migrate
[2018-11-30 16:29:44,505] INFO sign:16021-1543566578.479 lineno:64 : do register to 127.0.0.1,remainder:[]
[2018-11-30 16:29:44,509] INFO sign:16021-1543566578.479 lineno:93 : do register result:{u'code': 200, u'data': {u'pieceSize': 4194304, u'fileLength': -1, u'taskId': u'e602b9db61726ed79e4156a932fc0fa66d9a5958082736af3b1f94f0779e6f31'}} and cost 0.004
[2018-11-30 16:29:44,515] ERROR sign:16021-1543566578.479 lineno:119 : pull piece task fail:{u'msg': u'cdn status is fail', u'code': 604} and will migrate
[2018-11-30 16:29:44,515] ERROR sign:16021-1543566578.479 lineno:243 : p2p fail
Traceback (most recent call last):
  File "/usr/local/bin/core/fetcher.py", line 230, in run
    self.identifier)
  File "/usr/local/bin/component/httputil.py", line 132, in pull_piece_task
    http_path, md5, identifier)
  File "/usr/local/bin/component/httputil.py", line 126, in pull_piece_task
    identifier)
  File "/usr/local/bin/component/httputil.py", line 86, in register
    raise Exception("register result:%s" % result)
Exception: register result:None
[2018-11-30 16:29:44,516] INFO sign:16021-1543566578.479 lineno:604 : start download logo.png from the source station
[2018-11-30 16:29:45,429] INFO sign:16021-1543566578.479 lineno:109 : move src:/tmp/tmphGtb0T.backsource to dst:/tmp/logo.png result:True cost:0.000
[2018-11-30 16:29:45,455] INFO sign:16021-1543566578.479 lineno:60 : local http result:success for path:/client/ and cost:0.002
[2018-11-30 16:29:45,455] INFO sign:16021-1543566578.479 lineno:94 : |e602b9db61726ed79e4156a932fc0fa66d9a5958082736af3b1f94f0779e6f31|https://github.com/dragonflyoss/Dragonfly/blob/master/do/images/logo.png|129202|0|127.0.0.1|UNKNOWN|6.975|
[2018-11-30 16:29:45,456] INFO sign:16021-1543566578.479 lineno:111 : download SUCCESS cost:6.977s length:129202 reason:3

BlueBlue-Lee · 2018-11-30T09:51:38Z

Another question:

I deploy superNode in 10.173.32.47.

And then I deploy dfdaemon and dfget in 10.173.32.48:

my configuration:

#
/usr/bin/dockerd -H fd:// --registry-mirror http://127.0.0.1:65001

# nohup dfdaemon --registry hub.c.163.com --verbose &

# cat /etc/dragonfly.conf 
[node]
address=10.173.32.47

Then I docker pull images:

# docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
5de4b4d551f8: Pulling fs layer 
d4b36a5e9443: Pulling fs layer 
0af1f0713557: Retrying in 1 second 
^C

Check dfclient.log , lots of errors:

error: [Errno 111] Connection refused
[2018-11-30 17:38:50,973] ERROR sign:13283-1543570469.268 lineno:34 : connect to ip:127.0.0.1 port:8001 fail
Traceback (most recent call last):
  File "/usr/local/bin/component/netutil.py", line 31, in check_connect
    s.connect((ip, port))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[2018-11-30 17:38:50,976] ERROR sign:13283-1543570469.268 lineno:34 : connect to ip:127.0.0.1 port:8001 fail
Traceback (most recent call last):
  File "/usr/local/bin/component/netutil.py", line 31, in check_connect
    s.connect((ip, port))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[2018-11-30 17:38:50,979] ERROR sign:13283-1543570469.268 lineno:34 : connect to ip:127.0.0.1 port:8001 fail
Traceback (most recent call last):
  File "/usr/local/bin/component/netutil.py", line 31, in check_connect
    s.connect((ip, port))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[2018-11-30 17:38:50,982] ERROR sign:13283-1543570469.268 lineno:34 : connect to ip:127.0.0.1 port:8001 fail

I don't know why dfget willl conect to 127.0.0.1:8001. I have configure it in /etc/dragonfly.conf with:

# cat /etc/dragonfly.conf 
[node]
address=10.173.32.47

BlueBlue-Lee · 2018-12-03T09:35:21Z

For now, downloading file with dfget works fine, it can save lots of time benefitting from superNode cdn function:

# time wget -q https://dl.google.com/go/go1.10.5.freebsd-amd64.tar.gz
real	1m16.272s
user	0m0.780s
sys	0m0.628s

# time dfget -u 'https://dl.google.com/go/go1.10.5.freebsd-amd64.tar.gz' -o /tmp/go1.12.linux-adm64.tar.gz
--2018-12-03 17:27:23--  https://dl.google.com/go/go1.10.5.freebsd-amd64.tar.gz
current user[root] output path[/tmp/go1.12.linux-adm64.tar.gz]
dfget version:0.2.0
workspace:/root/.small-dragonfly/ sign:47701-1543829243.698
client:10.173.32.49 connected to node:10.173.32.47
init success...
start download by dragonfly
download SUCCESS(0) cost(4.733s) length:88123613 reason:0

real	0m4.821s
user	0m0.524s
sys	0m0.300s

Kindly remind， this file(https://github.com/dragonflyoss/Dragonfly/blob/master/do/images/logo.png) is already gone in quick start doc

allencloud · 2018-12-05T03:29:25Z

@starnop Could you help to take a look at of this?
To see if we could reproduce this issue according to @shenlanse 's steps.

Once find the root cause, we could take action to backport the fix or fix in the new refactored releases.

BlueBlue-Lee · 2018-12-05T10:36:16Z

@allencloud thanks for your reply. 好吧还是说中文吧，不然效率有点低。

之前我的虚拟机的网络是经典网络的，部分镜像是能够正常pull下来的，但是大概率pull镜像是失败的，包括client那边的原因比如register失败，p2p fail或者 607 invalid url，以及superNode那边的原因比如connect error，read piece time out等。

后来我重建了几台虚拟机用的是vpc网络，好吧，问题基本消失了，基本上都能够成功，只有少数case会失败。

看起来是我这边网络环境的问题，不过另一方面也希望社区能够不断完善，不要对网络这么脆弱敏感（因为我不用蜻蜓的情况下，docker pull任何镜像都是正常的）。

BlueBlue-Lee · 2018-12-05T10:39:18Z

还有我这边整理了dragonfly的
部署文档，包括使用deployment部署superNode，使用daemonset部署dfget和dfdaemon等。

后面我会整理一下提交到社区

starnop · 2018-12-05T15:54:24Z

还有我这边整理了dragonfly的
部署文档，包括使用deployment部署superNode，使用daemonset部署dfget和dfdaemon等。

后面我会整理一下提交到社区

In fact, we plan to support the deployment of dragonfly with helm, and the repo is here. And � a pull request has been submitted, welcome to put forward your opinions, let's make it perfect together. ^_^

BlueBlue-Lee · 2018-12-06T01:51:18Z

@starnop Good, get it.

lowzj · 2018-12-19T08:40:31Z

I don't know why dfget will connect to 127.0.0.1:8001. I have configure it in /etc/dragonfly.conf with:
# cat /etc/dragonfly.conf 
[node]
address=10.173.32.47

@shenlanse
The workaround solution is to restart the supernode.
The reason is that supernode will set its own local ip as the value which the client connect it via at the first time.

lowzj · 2018-12-19T08:45:43Z

还有我这边整理了dragonfly的
部署文档，包括使用deployment部署superNode，使用daemonset部署dfget和dfdaemon等。

后面我会整理一下提交到社区

We have blog section in our dragonfly website:

English: https://d7y.io/en-us/blog/index.html
Chinese: https://d7y.io/zh-cn/blog/index.html

@shenlanse Could we post your article on https://d7y.io ?

BlueBlue-Lee · 2018-12-19T09:51:47Z

还有我这边整理了dragonfly的
部署文档，包括使用deployment部署superNode，使用daemonset部署dfget和dfdaemon等。
后面我会整理一下提交到社区

We have blog section in our dragonfly website:

English: https://d7y.io/en-us/blog/index.html

Chinese: https://d7y.io/zh-cn/blog/index.html

@shenlanse Could we post your article on https://d7y.io ?

@lowzj Of course, up to you.

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj · 2018-12-24T08:21:49Z

@shenlanse Thanks a lot ^-^, I have post your document: https://d7y.io/zh-cn/blog/deploy-dragonfly-on-k8s.html

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

allencloud · 2019-01-07T16:05:20Z

First of all, thanks sincerely for constantly using and supporting Dragonfly. We will try our best to keep Dragonfly better, and keep community and eco-system growing.

To get more feedback of Dragonfly's adoption in industry, would you mind helping comment in issue dragonflyoss/dragonfly#219 to give more information about your enterprise's Dragonfly usage. Thanks a lot in advance. @shenlanse

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

* test: create cache Signed-off-by: Gaius <[email protected]> * feat: error check Signed-off-by: Gaius <[email protected]>

pouchrobot added the kind/question all questions or confusion about this project label Nov 30, 2018

lowzj added a commit to lowzj/Dragonfly that referenced this issue Dec 24, 2018

fix: unreachable supernode ip in p2p-network

f382f0c

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj mentioned this issue Dec 24, 2018

fix: unreachable supernode ip in p2p-network #277

Merged

lowzj added a commit to lowzj/Dragonfly that referenced this issue Dec 25, 2018

fix: unreachable supernode ip in p2p-network

bbad84b

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj added a commit to lowzj/Dragonfly that referenced this issue Jan 31, 2019

fix: unreachable supernode ip in p2p-network

343d417

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj added a commit to lowzj/Dragonfly that referenced this issue Jan 31, 2019

fix: unreachable supernode ip in p2p-network

b4fa21b

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj added a commit to lowzj/Dragonfly that referenced this issue Feb 2, 2019

fix: unreachable supernode ip in p2p-network

9ac9950

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj added a commit to lowzj/Dragonfly that referenced this issue Feb 2, 2019

fix: unreachable supernode ip in p2p-network

4ea9917

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj added a commit to lowzj/Dragonfly that referenced this issue Mar 1, 2019

fix: unreachable supernode ip in p2p-network

5b3f004

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj added a commit to lowzj/Dragonfly that referenced this issue Mar 1, 2019

fix: unreachable supernode ip in p2p-network

0d29e3f

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

lowzj added a commit to lowzj/Dragonfly that referenced this issue Mar 1, 2019

fix: unreachable supernode ip in p2p-network

432bd86

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

starnop closed this as completed in #277 Mar 1, 2019

starnop pushed a commit to starnop/Dragonfly that referenced this issue Nov 27, 2019

fix: unreachable supernode ip in p2p-network

9d1ac91

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

inoc603 pushed a commit to inoc603/Dragonfly that referenced this issue Dec 23, 2019

fix: unreachable supernode ip in p2p-network

80a1f56

see issue: dragonflyoss#252 Signed-off-by: lowzj <[email protected]>

sungjunyoung pushed a commit to sungjunyoung/Dragonfly that referenced this issue May 8, 2022

Fix scheduler test (dragonflyoss#252)

b600800

* test: create cache Signed-off-by: Gaius <[email protected]> * feat: error check Signed-off-by: Gaius <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Question] dfget fails to pull images #252

[Question] dfget fails to pull images #252

BlueBlue-Lee commented Nov 30, 2018

BlueBlue-Lee commented Nov 30, 2018 •

edited

Loading

BlueBlue-Lee commented Nov 30, 2018

BlueBlue-Lee commented Nov 30, 2018

BlueBlue-Lee commented Nov 30, 2018

BlueBlue-Lee commented Dec 3, 2018

allencloud commented Dec 5, 2018

BlueBlue-Lee commented Dec 5, 2018

BlueBlue-Lee commented Dec 5, 2018

starnop commented Dec 5, 2018 •

edited

Loading

BlueBlue-Lee commented Dec 6, 2018

lowzj commented Dec 19, 2018 •

edited

Loading

lowzj commented Dec 19, 2018

BlueBlue-Lee commented Dec 19, 2018 •

edited

Loading

lowzj commented Dec 24, 2018

allencloud commented Jan 7, 2019 •

edited

Loading

[Question] dfget fails to pull images #252

[Question] dfget fails to pull images #252

Comments

BlueBlue-Lee commented Nov 30, 2018

Question

BlueBlue-Lee commented Nov 30, 2018 • edited Loading

BlueBlue-Lee commented Nov 30, 2018

BlueBlue-Lee commented Nov 30, 2018

BlueBlue-Lee commented Nov 30, 2018

BlueBlue-Lee commented Dec 3, 2018

allencloud commented Dec 5, 2018

BlueBlue-Lee commented Dec 5, 2018

BlueBlue-Lee commented Dec 5, 2018

starnop commented Dec 5, 2018 • edited Loading

BlueBlue-Lee commented Dec 6, 2018

lowzj commented Dec 19, 2018 • edited Loading

lowzj commented Dec 19, 2018

BlueBlue-Lee commented Dec 19, 2018 • edited Loading

lowzj commented Dec 24, 2018

allencloud commented Jan 7, 2019 • edited Loading

BlueBlue-Lee commented Nov 30, 2018 •

edited

Loading

starnop commented Dec 5, 2018 •

edited

Loading

lowzj commented Dec 19, 2018 •

edited

Loading

BlueBlue-Lee commented Dec 19, 2018 •

edited

Loading

allencloud commented Jan 7, 2019 •

edited

Loading