Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X...X.X.] duration:
6.041223629s trace:
https://jaeger.dev.storj.io/trace/199936ee310c722d
FAIL (2025-12-29 15:08:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(575h38m5.169319486s) [upload=504612]: Uploading segments...
Dec 29 15:07:05.298473852 (575h38m5.169556888s) [upload=504612]: Got next segment. Inline: false
Dec 29 15:07:05.298509651 (575h38m5.169592686s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:07:05.298514481 (575h38m5.169597526s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:07:05.547270627 (575h38m5.418353672s) [seg_pos={0 0}, upload=504612]: Begin upload segment...
Dec 29 15:07:05.547358023 (575h38m5.418441068s) [seg_pos={0 0}, upload=504612]: Done begin upload segment.
Dec 29 15:07:05.547399332 (575h38m5.418482377s) [noise=true , address=145.79.227.8:4331 , piece_id=HJMDCXGTRVQSWXJFYYZUM32X6HOVJOALWAA77PMO7DHSVGKDSQOQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.547738418 (575h38m5.418821463s) [noise=true , address=145.79.227.37:4565 , piece_id=2J5SVYAME2ZIZ4KL3YGFSIE6XM4ZBPUB5C6BMUQB7I5CMDWGOILQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.548643221 (575h38m5.419726286s) [noise=true , address=145.79.227.15:4876 , piece_id=PLIQKHXY3XX7ZQNBJIMZI47XI463JJPRH457PPHIK6LX5XJZYDKQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.549116953 (575h38m5.420199988s) [noise=true , address=45.91.116.7:4009 , piece_id=HRFUZ647EIJDJHL3RN7IA32UDBLKX4M4V6HUE4F5TRQNPOWQDJWQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.549739037 (575h38m5.420822072s) [noise=true , address=145.79.227.11:4730 , piece_id=N4NTQQ66D5D4BZLYVVAA5N2PFLFV6KA6ONICZMGV64N4DS47TYCA, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.550335674 (575h38m5.421418749s) [noise=true , address=145.79.227.3:4149 , piece_id=CEVXKR5W34XCR3Q6GAGFHG2VS46LMOK4BRPD6WBHHR6GFTKSKX3Q, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.550852457 (575h38m5.421935532s) [noise=true , address=77.90.31.11:4553 , piece_id=TTZT5JMQNTRPMQZ5JPVBCTYBHNIJ2ZSEHJ5KPLGS52I7IJBW25WQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.552153264 (575h38m5.42323631s) [noise=true , address=145.79.227.6:4258 , piece_id=G47JVZFATLN3FKNQ76DTRLFAIDP3YS6P4KPCYD4TZYDG6GBJCEEQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.552436794 (575h38m5.423519839s) [noise=true , address=145.79.227.17:4972 , piece_id=YL3OJZS4MU2FN4YB4CDZZPYFTZJSHL5UVQZ6PAJZQ2KYZZTZYYMQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.553505570 (575h38m5.424588615s) [noise=true , address=45.91.116.5:4297 , piece_id=E2V6G5UW4K7AO2BGB6DASPN4VCGI2FMQPLYWNEAF3ICVW7FLT3BQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.554885296 (575h38m5.425968331s) [noise=true , address=145.79.227.48:4302 , piece_id=GNJ4PG2LPKGHM4ZPZPQ4URHAZFDY5ZENM6GDNPL6GWJSMWULE55Q, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.555987936 (575h38m5.427070971s) [noise=true , address=45.91.116.11:4118 , piece_id=NEXPAWSRLQJFMOXS2MIP7HNTLZADAUOTHZHWH22ZDP25XD7ARJ6A, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.557090746 (575h38m5.428173782s) [noise=true , address=145.79.227.27:4690 , piece_id=YZNO6NCESTJ3IC2NUWU3HI4BS34LVV3ZAKWX7CPZVVAM2TUOMH5Q, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.558225968 (575h38m5.429309013s) [noise=true , address=77.90.31.10:4519 , piece_id=INRLGMFVECEI56CZYKIYQ27QLCEKVETIHEUH42XMOU5OJ4VNDHOA, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.559022384 (575h38m5.430105429s) [noise=true , address=145.79.227.36:4519 , piece_id=GHHQJSG6IONCZXE2B4LKQ2YXDMSXFBUXJVRM72RXGMBTRL5OMUJQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.559773204 (575h38m5.430856239s) [noise=true , address=45.91.116.8:4048 , piece_id=V56UFHJGH3JEUZYZJHPDQT6D2VA4KKI3QUI6FM37YOIUKE2GUYOA, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:07.258028016 (575h38m7.129111061s) [upload=504612]: Next returned nil segment
Dec 29 15:07:07.258043756 (575h38m7.129126801s) [upload=504612]: Waiting for error group managing segments...
Dec 29 15:07:07.534717474 (575h38m7.40580052s) [noise=true , address=145.79.227.37:4565 , piece_id=2J5SVYAME2ZIZ4KL3YGFSIE6XM4ZBPUB5C6BMUQB7I5CMDWGOILQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.536820148 (575h38m7.407903183s) [noise=true , address=145.79.227.11:4730 , piece_id=N4NTQQ66D5D4BZLYVVAA5N2PFLFV6KA6ONICZMGV64N4DS47TYCA, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.544881918 (575h38m7.415964963s) [noise=true , address=145.79.227.6:4258 , piece_id=G47JVZFATLN3FKNQ76DTRLFAIDP3YS6P4KPCYD4TZYDG6GBJCEEQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.546641349 (575h38m7.417724394s) [noise=true , address=145.79.227.17:4972 , piece_id=YL3OJZS4MU2FN4YB4CDZZPYFTZJSHL5UVQZ6PAJZQ2KYZZTZYYMQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.546948514 (575h38m7.418031559s) [noise=true , address=145.79.227.48:4302 , piece_id=GNJ4PG2LPKGHM4ZPZPQ4URHAZFDY5ZENM6GDNPL6GWJSMWULE55Q, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.559719682 (575h38m7.430802717s) [noise=true , address=145.79.227.15:4876 , piece_id=PLIQKHXY3XX7ZQNBJIMZI47XI463JJPRH457PPHIK6LX5XJZYDKQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.565921601 (575h38m7.437004636s) [noise=true , address=145.79.227.3:4149 , piece_id=CEVXKR5W34XCR3Q6GAGFHG2VS46LMOK4BRPD6WBHHR6GFTKSKX3Q, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.565945978 (575h38m7.437029023s) [noise=true , address=145.79.227.27:4690 , piece_id=YZNO6NCESTJ3IC2NUWU3HI4BS34LVV3ZAKWX7CPZVVAM2TUOMH5Q, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.572085899 (575h38m7.443168964s) [noise=true , address=145.79.227.8:4331 , piece_id=HJMDCXGTRVQSWXJFYYZUM32X6HOVJOALWAA77PMO7DHSVGKDSQOQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.581407739 (575h38m7.452490785s) [noise=true , address=145.79.227.36:4519 , piece_id=GHHQJSG6IONCZXE2B4LKQ2YXDMSXFBUXJVRM72RXGMBTRL5OMUJQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.759953628 (575h38m7.631036673s) [noise=true , address=77.90.31.10:4519 , piece_id=INRLGMFVECEI56CZYKIYQ27QLCEKVETIHEUH42XMOU5OJ4VNDHOA, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.814297658 (575h38m7.685380703s) [noise=true , address=45.91.116.11:4118 , piece_id=NEXPAWSRLQJFMOXS2MIP7HNTLZADAUOTHZHWH22ZDP25XD7ARJ6A, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.846747206 (575h38m7.717830251s) [noise=true , address=45.91.116.5:4297 , piece_id=E2V6G5UW4K7AO2BGB6DASPN4VCGI2FMQPLYWNEAF3ICVW7FLT3BQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.846813162 (575h38m7.717896207s) [seg_pos={0 0}, upload=504612]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:07:07.847039543 (575h38m7.718122578s) [noise=true , address=45.91.116.7:4009 , piece_id=HRFUZ647EIJDJHL3RN7IA32UDBLKX4M4V6HUE4F5TRQNPOWQDJWQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:37142->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:37142->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37142->45.91.116.7:4009: use of closed network connection
Dec 29 15:07:07.847048330 (575h38m7.718131385s) [noise=true , address=45.91.116.7:4009 , piece_id=HRFUZ647EIJDJHL3RN7IA32UDBLKX4M4V6HUE4F5TRQNPOWQDJWQ, seg_pos={0 0}, upload=504612]: Long tail cancellation due to optimal threshold
Dec 29 15:07:07.847233041 (575h38m7.718316087s) [noise=true , address=77.90.31.11:4553 , piece_id=TTZT5JMQNTRPMQZ5JPVBCTYBHNIJ2ZSEHJ5KPLGS52I7IJBW25WQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF
Dec 29 15:07:07.847254121 (575h38m7.718337166s) [noise=true , address=77.90.31.11:4553 , piece_id=TTZT5JMQNTRPMQZ5JPVBCTYBHNIJ2ZSEHJ5KPLGS52I7IJBW25WQ, seg_pos={0 0}, upload=504612]: Long tail cancellation due to optimal threshold
Dec 29 15:07:07.847334855 (575h38m7.7184179s) [noise=true , address=45.91.116.8:4048 , piece_id=V56UFHJGH3JEUZYZJHPDQT6D2VA4KKI3QUI6FM37YOIUKE2GUYOA, seg_pos={0 0}, upload=504612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:45732->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:45732->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45732->45.91.116.8:4048: use of closed network connection
Dec 29 15:07:07.847346638 (575h38m7.718429673s) [noise=true , address=45.91.116.8:4048 , piece_id=V56UFHJGH3JEUZYZJHPDQT6D2VA4KKI3QUI6FM37YOIUKE2GUYOA, seg_pos={0 0}, upload=504612]: Long tail cancellation due to optimal threshold
Dec 29 15:07:07.847393877 (575h38m7.718476922s) [seg_pos={0 0}, upload=504612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:07:07.847510740 (575h38m7.718593785s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:07:07.847525498 (575h38m7.718608553s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:07:08.149479063 (575h38m8.020562098s) [upload=504612]: Done uploading segments...
Dec 29 15:07:08.149500182 (575h38m8.020583227s) [upload=504612]: Upload finished. err: <nil>
Duration: 6.041268123s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
7.268939131s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:08:04 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 7.268960963s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...XX.X.XXXXX.X..X.....X....X....X...XX.............X.....X.] duration:
25.700524669s trace:
https://jaeger.dev.storj.io/trace/2f83f4b9673d2f1c
FAIL (2025-12-29 15:08:01 UTC)
orbiter warning: context deadline exceeded
Duration: 25.700690604s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXX.X.] duration:
3.718030939s trace:
https://jaeger.dev.storj.io/trace/65c64dd6931c5cc
FAIL (2025-12-29 15:07:59 UTC)
orbiter warning: context deadline exceeded
Duration: 3.718057419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..................XXX...........XX.X........................] duration:
20.534674331s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:07:50 UTC)
orbiter warning: context deadline exceeded
Duration: 20.534796865s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
10.743875167s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:07:41 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.743910785s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X...X.X] duration:
30.00367945s trace:
https://jaeger.dev.storj.io/trace/199936ee310c722d
FAIL (2025-12-29 15:07:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(575h38m5.169319486s) [upload=504612]: Uploading segments...
Dec 29 15:07:05.298473852 (575h38m5.169556888s) [upload=504612]: Got next segment. Inline: false
Dec 29 15:07:05.298509651 (575h38m5.169592686s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:07:05.298514481 (575h38m5.169597526s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:07:05.547270627 (575h38m5.418353672s) [seg_pos={0 0}, upload=504612]: Begin upload segment...
Dec 29 15:07:05.547358023 (575h38m5.418441068s) [seg_pos={0 0}, upload=504612]: Done begin upload segment.
Dec 29 15:07:05.547399332 (575h38m5.418482377s) [noise=true , address=145.79.227.8:4331 , piece_id=HJMDCXGTRVQSWXJFYYZUM32X6HOVJOALWAA77PMO7DHSVGKDSQOQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.547738418 (575h38m5.418821463s) [noise=true , address=145.79.227.37:4565 , piece_id=2J5SVYAME2ZIZ4KL3YGFSIE6XM4ZBPUB5C6BMUQB7I5CMDWGOILQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.548643221 (575h38m5.419726286s) [noise=true , address=145.79.227.15:4876 , piece_id=PLIQKHXY3XX7ZQNBJIMZI47XI463JJPRH457PPHIK6LX5XJZYDKQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.549116953 (575h38m5.420199988s) [noise=true , address=45.91.116.7:4009 , piece_id=HRFUZ647EIJDJHL3RN7IA32UDBLKX4M4V6HUE4F5TRQNPOWQDJWQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.549739037 (575h38m5.420822072s) [noise=true , address=145.79.227.11:4730 , piece_id=N4NTQQ66D5D4BZLYVVAA5N2PFLFV6KA6ONICZMGV64N4DS47TYCA, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.550335674 (575h38m5.421418749s) [noise=true , address=145.79.227.3:4149 , piece_id=CEVXKR5W34XCR3Q6GAGFHG2VS46LMOK4BRPD6WBHHR6GFTKSKX3Q, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.550852457 (575h38m5.421935532s) [noise=true , address=77.90.31.11:4553 , piece_id=TTZT5JMQNTRPMQZ5JPVBCTYBHNIJ2ZSEHJ5KPLGS52I7IJBW25WQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.552153264 (575h38m5.42323631s) [noise=true , address=145.79.227.6:4258 , piece_id=G47JVZFATLN3FKNQ76DTRLFAIDP3YS6P4KPCYD4TZYDG6GBJCEEQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.552436794 (575h38m5.423519839s) [noise=true , address=145.79.227.17:4972 , piece_id=YL3OJZS4MU2FN4YB4CDZZPYFTZJSHL5UVQZ6PAJZQ2KYZZTZYYMQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.553505570 (575h38m5.424588615s) [noise=true , address=45.91.116.5:4297 , piece_id=E2V6G5UW4K7AO2BGB6DASPN4VCGI2FMQPLYWNEAF3ICVW7FLT3BQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.554885296 (575h38m5.425968331s) [noise=true , address=145.79.227.48:4302 , piece_id=GNJ4PG2LPKGHM4ZPZPQ4URHAZFDY5ZENM6GDNPL6GWJSMWULE55Q, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.555987936 (575h38m5.427070971s) [noise=true , address=45.91.116.11:4118 , piece_id=NEXPAWSRLQJFMOXS2MIP7HNTLZADAUOTHZHWH22ZDP25XD7ARJ6A, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.557090746 (575h38m5.428173782s) [noise=true , address=145.79.227.27:4690 , piece_id=YZNO6NCESTJ3IC2NUWU3HI4BS34LVV3ZAKWX7CPZVVAM2TUOMH5Q, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.558225968 (575h38m5.429309013s) [noise=true , address=77.90.31.10:4519 , piece_id=INRLGMFVECEI56CZYKIYQ27QLCEKVETIHEUH42XMOU5OJ4VNDHOA, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.559022384 (575h38m5.430105429s) [noise=true , address=145.79.227.36:4519 , piece_id=GHHQJSG6IONCZXE2B4LKQ2YXDMSXFBUXJVRM72RXGMBTRL5OMUJQ, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:05.559773204 (575h38m5.430856239s) [noise=true , address=45.91.116.8:4048 , piece_id=V56UFHJGH3JEUZYZJHPDQT6D2VA4KKI3QUI6FM37YOIUKE2GUYOA, seg_pos={0 0}, upload=504612]: Uploading piece...
Dec 29 15:07:07.258028016 (575h38m7.129111061s) [upload=504612]: Next returned nil segment
Dec 29 15:07:07.258043756 (575h38m7.129126801s) [upload=504612]: Waiting for error group managing segments...
Dec 29 15:07:07.534717474 (575h38m7.40580052s) [noise=true , address=145.79.227.37:4565 , piece_id=2J5SVYAME2ZIZ4KL3YGFSIE6XM4ZBPUB5C6BMUQB7I5CMDWGOILQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.536820148 (575h38m7.407903183s) [noise=true , address=145.79.227.11:4730 , piece_id=N4NTQQ66D5D4BZLYVVAA5N2PFLFV6KA6ONICZMGV64N4DS47TYCA, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.544881918 (575h38m7.415964963s) [noise=true , address=145.79.227.6:4258 , piece_id=G47JVZFATLN3FKNQ76DTRLFAIDP3YS6P4KPCYD4TZYDG6GBJCEEQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.546641349 (575h38m7.417724394s) [noise=true , address=145.79.227.17:4972 , piece_id=YL3OJZS4MU2FN4YB4CDZZPYFTZJSHL5UVQZ6PAJZQ2KYZZTZYYMQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.546948514 (575h38m7.418031559s) [noise=true , address=145.79.227.48:4302 , piece_id=GNJ4PG2LPKGHM4ZPZPQ4URHAZFDY5ZENM6GDNPL6GWJSMWULE55Q, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.559719682 (575h38m7.430802717s) [noise=true , address=145.79.227.15:4876 , piece_id=PLIQKHXY3XX7ZQNBJIMZI47XI463JJPRH457PPHIK6LX5XJZYDKQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.565921601 (575h38m7.437004636s) [noise=true , address=145.79.227.3:4149 , piece_id=CEVXKR5W34XCR3Q6GAGFHG2VS46LMOK4BRPD6WBHHR6GFTKSKX3Q, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.565945978 (575h38m7.437029023s) [noise=true , address=145.79.227.27:4690 , piece_id=YZNO6NCESTJ3IC2NUWU3HI4BS34LVV3ZAKWX7CPZVVAM2TUOMH5Q, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.572085899 (575h38m7.443168964s) [noise=true , address=145.79.227.8:4331 , piece_id=HJMDCXGTRVQSWXJFYYZUM32X6HOVJOALWAA77PMO7DHSVGKDSQOQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.581407739 (575h38m7.452490785s) [noise=true , address=145.79.227.36:4519 , piece_id=GHHQJSG6IONCZXE2B4LKQ2YXDMSXFBUXJVRM72RXGMBTRL5OMUJQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.759953628 (575h38m7.631036673s) [noise=true , address=77.90.31.10:4519 , piece_id=INRLGMFVECEI56CZYKIYQ27QLCEKVETIHEUH42XMOU5OJ4VNDHOA, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.814297658 (575h38m7.685380703s) [noise=true , address=45.91.116.11:4118 , piece_id=NEXPAWSRLQJFMOXS2MIP7HNTLZADAUOTHZHWH22ZDP25XD7ARJ6A, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.846747206 (575h38m7.717830251s) [noise=true , address=45.91.116.5:4297 , piece_id=E2V6G5UW4K7AO2BGB6DASPN4VCGI2FMQPLYWNEAF3ICVW7FLT3BQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: <nil>
Dec 29 15:07:07.846813162 (575h38m7.717896207s) [seg_pos={0 0}, upload=504612]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:07:07.847039543 (575h38m7.718122578s) [noise=true , address=45.91.116.7:4009 , piece_id=HRFUZ647EIJDJHL3RN7IA32UDBLKX4M4V6HUE4F5TRQNPOWQDJWQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:37142->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:37142->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37142->45.91.116.7:4009: use of closed network connection
Dec 29 15:07:07.847048330 (575h38m7.718131385s) [noise=true , address=45.91.116.7:4009 , piece_id=HRFUZ647EIJDJHL3RN7IA32UDBLKX4M4V6HUE4F5TRQNPOWQDJWQ, seg_pos={0 0}, upload=504612]: Long tail cancellation due to optimal threshold
Dec 29 15:07:07.847233041 (575h38m7.718316087s) [noise=true , address=77.90.31.11:4553 , piece_id=TTZT5JMQNTRPMQZ5JPVBCTYBHNIJ2ZSEHJ5KPLGS52I7IJBW25WQ, seg_pos={0 0}, upload=504612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF
Dec 29 15:07:07.847254121 (575h38m7.718337166s) [noise=true , address=77.90.31.11:4553 , piece_id=TTZT5JMQNTRPMQZ5JPVBCTYBHNIJ2ZSEHJ5KPLGS52I7IJBW25WQ, seg_pos={0 0}, upload=504612]: Long tail cancellation due to optimal threshold
Dec 29 15:07:07.847334855 (575h38m7.7184179s) [noise=true , address=45.91.116.8:4048 , piece_id=V56UFHJGH3JEUZYZJHPDQT6D2VA4KKI3QUI6FM37YOIUKE2GUYOA, seg_pos={0 0}, upload=504612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:45732->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:45732->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45732->45.91.116.8:4048: use of closed network connection
Dec 29 15:07:07.847346638 (575h38m7.718429673s) [noise=true , address=45.91.116.8:4048 , piece_id=V56UFHJGH3JEUZYZJHPDQT6D2VA4KKI3QUI6FM37YOIUKE2GUYOA, seg_pos={0 0}, upload=504612]: Long tail cancellation due to optimal threshold
Dec 29 15:07:07.847393877 (575h38m7.718476922s) [seg_pos={0 0}, upload=504612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:07:07.847510740 (575h38m7.718593785s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:07:07.847525498 (575h38m7.718608553s) [upload=504612]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:07:08.149479063 (575h38m8.020562098s) [upload=504612]: Done uploading segments...
Dec 29 15:07:08.149500182 (575h38m8.020583227s) [upload=504612]: Upload finished. err: <nil>
Duration: 30.003694769s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXX.X] duration:
5.000579161s trace:
https://jaeger.dev.storj.io/trace/65c64dd6931c5cc
FAIL (2025-12-29 15:07:28 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000590773s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X...XX.X.XXXXX.X..X.....X....X....X...XX.............X.....X] duration:
30.000853337s trace:
https://jaeger.dev.storj.io/trace/2f83f4b9673d2f1c
FAIL (2025-12-29 15:07:26 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000863115s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [............X........X.........X......X.......X...X.........] duration:
44.525219274s trace:
https://jaeger.dev.storj.io/trace/24cdf967807c73aa
FAIL (2025-12-29 15:07:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded
======LOGS======
575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment
Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments...
Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection
Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments...
Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil>
Duration: 44.525234032s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...................XXX...........XX.X.......................] duration:
19.969992443s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:06:59 UTC)
orbiter warning: context deadline exceeded
Duration: 19.970050705s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XX...XX.X.XXXXX.X..X.....X....X....X...XX.............X.....] duration:
23.361844623s trace:
https://jaeger.dev.storj.io/trace/2497001198689567
FAIL (2025-12-29 15:06:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.361861425s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
8.660322085s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:06:51 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.660348194s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............X........X.........X......X.......X...X........] duration:
43.155947157s trace:
https://jaeger.dev.storj.io/trace/24cdf967807c73aa
FAIL (2025-12-29 15:06:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded
======LOGS======
575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment
Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments...
Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection
Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments...
Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil>
Duration: 43.155972886s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X...X.] duration:
6.209230428s trace:
https://jaeger.dev.storj.io/trace/2df80b1c0a7a2f74
FAIL (2025-12-29 15:06:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: HHH7Y3T63A7X3CBEMGOE6RRGAT3NEC3MUGWOUHIXJSSNQJV6WXNQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: Z5LVU3A4BDG3N64TQ7EMEDKZDUCNAQERNJHRGNPEAW6PCTJICA4A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: YN3RPHA2GTMUV3QMERHZZQH477I5L3TEOMV2OEE4N4K33AFANMLQ) context deadline exceeded
======LOGS======
Dec 29 15:05:33.320756751 (575h36m33.191839796s) [upload=504590]: Starting upload
Dec 29 15:05:33.320843436 (575h36m33.191926471s) [upload=504590]: Done starting upload
Dec 29 15:05:33.321015714 (575h36m33.192098749s) [upload=504590]: Uploading segments...
Dec 29 15:05:33.321109022 (575h36m33.192192067s) [upload=504590]: Got next segment. Inline: false
Dec 29 15:05:33.321253677 (575h36m33.192336722s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:05:33.321356844 (575h36m33.192439879s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:05:33.571619792 (575h36m33.442702837s) [seg_pos={0 0}, upload=504590]: Begin upload segment...
Dec 29 15:05:33.571700706 (575h36m33.442783751s) [seg_pos={0 0}, upload=504590]: Done begin upload segment.
Dec 29 15:05:33.571735212 (575h36m33.442818257s) [noise=true , address=145.79.227.35:4483 , piece_id=MSHSUZCFEVIKOSNJCSK2NZTZHTKWIKAHQZDZQP54XXE5DLPTWHXA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.571763375 (575h36m33.442846421s) [noise=true , address=45.91.116.13:4153 , piece_id=YN3RPHA2GTMUV3QMERHZZQH477I5L3TEOMV2OEE4N4K33AFANMLQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.571823270 (575h36m33.442906305s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOMPU2T5XSFDDCVDKWVMI6J4HCSV6IT5TVVNN5FEE4C4KQRWNS3Q, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.572381743 (575h36m33.443464778s) [noise=true , address=45.91.116.9:4263 , piece_id=HHH7Y3T63A7X3CBEMGOE6RRGAT3NEC3MUGWOUHIXJSSNQJV6WXNQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.572889009 (575h36m33.443972054s) [noise=true , address=45.91.116.6:4973 , piece_id=KMVPCXA5TU3G6MGD2E3JBTE6J25WRALAFPP7ZQERKDKWX3MWVOYQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.575125186 (575h36m33.446208231s) [noise=true , address=45.91.116.14:4189 , piece_id=Z5LVU3A4BDG3N64TQ7EMEDKZDUCNAQERNJHRGNPEAW6PCTJICA4A, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.575948034 (575h36m33.447031069s) [noise=true , address=145.79.227.36:4517 , piece_id=LEFPRX4ZSDPOWPVKH5ZKOHPPKU72I2XDT3QFINXGQLKIIZQ5SELQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.576999195 (575h36m33.44808223s) [noise=true , address=77.90.31.15:4697 , piece_id=FWJAR6DWRS547CN6JM2ZJQI3IYMAMQXFEEA62IYZGF3IF7IF5IYQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.579632980 (575h36m33.450716016s) [noise=true , address=145.79.227.51:4410 , piece_id=7YL6EUDMHYJDE7TXHRGULLOSSLZF2NK2JW7Q5ZRCOCB5OYGMPHAA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.582358440 (575h36m33.453441485s) [noise=true , address=145.79.227.15:4874 , piece_id=STCGRECFZ4A7IC2MIIKHOJCXJ7ID4BDLHPY424T2P7EFLBI7QMGQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.583788403 (575h36m33.454871438s) [noise=true , address=145.79.227.28:4726 , piece_id=PDEIQUPWNI5SNB74TQDCV5P3IR7Y746HGI4WPPRL67LYMCJ2L2JA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.585498619 (575h36m33.456581664s) [noise=true , address=145.79.227.14:4838 , piece_id=V5GBHM3ZY2A645LQVMB2KVBTSNPX6EHEENNLIUE3VUJPRB7BDACA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.585763243 (575h36m33.456846279s) [noise=true , address=145.79.227.52:4447 , piece_id=EWXB6GEKWJMPCH3S4P6J3XWY7BB4ZV2L7P7ZTH3OCWCC5HMPSC2Q, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.587052338 (575h36m33.458135383s) [noise=true , address=145.79.227.6:4257 , piece_id=6U6WMIJTCIXG4WCBYHWCENZFNXBOD2IE2PPZK5U6BU7IEY6LAFKQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.587650046 (575h36m33.458733091s) [noise=true , address=145.79.227.17:4971 , piece_id=KW72WRRTAABLCIHKNETHTGHSOPQXPAJM2N7VVTUBVGCXG6P6BQPQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.588353686 (575h36m33.459436731s) [noise=true , address=145.79.227.7:4295 , piece_id=FIORUD3W4D3ZEY7H7CDKIZKQTBV2LKWXMQGTC4TCASAPHNCHTTAA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:35.280113881 (575h36m35.151196926s) [upload=504590]: Next returned nil segment
Dec 29 15:05:35.280149249 (575h36m35.151232304s) [upload=504590]: Waiting for error group managing segments...
Dec 29 15:05:35.563124777 (575h36m35.434207812s) [noise=true , address=145.79.227.28:4726 , piece_id=PDEIQUPWNI5SNB74TQDCV5P3IR7Y746HGI4WPPRL67LYMCJ2L2JA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.566350390 (575h36m35.437433425s) [noise=true , address=145.79.227.51:4410 , piece_id=7YL6EUDMHYJDE7TXHRGULLOSSLZF2NK2JW7Q5ZRCOCB5OYGMPHAA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.567063728 (575h36m35.438146763s) [noise=true , address=145.79.227.6:4257 , piece_id=6U6WMIJTCIXG4WCBYHWCENZFNXBOD2IE2PPZK5U6BU7IEY6LAFKQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.569454370 (575h36m35.440537405s) [noise=true , address=145.79.227.14:4838 , piece_id=V5GBHM3ZY2A645LQVMB2KVBTSNPX6EHEENNLIUE3VUJPRB7BDACA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.574928824 (575h36m35.446011869s) [noise=true , address=145.79.227.17:4971 , piece_id=KW72WRRTAABLCIHKNETHTGHSOPQXPAJM2N7VVTUBVGCXG6P6BQPQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.576093883 (575h36m35.447176918s) [noise=true , address=145.79.227.36:4517 , piece_id=LEFPRX4ZSDPOWPVKH5ZKOHPPKU72I2XDT3QFINXGQLKIIZQ5SELQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.580933889 (575h36m35.452016934s) [noise=true , address=145.79.227.52:4447 , piece_id=EWXB6GEKWJMPCH3S4P6J3XWY7BB4ZV2L7P7ZTH3OCWCC5HMPSC2Q, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.594457560 (575h36m35.465540605s) [noise=true , address=145.79.227.7:4295 , piece_id=FIORUD3W4D3ZEY7H7CDKIZKQTBV2LKWXMQGTC4TCASAPHNCHTTAA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.671937163 (575h36m35.543020208s) [noise=true , address=145.79.227.15:4874 , piece_id=STCGRECFZ4A7IC2MIIKHOJCXJ7ID4BDLHPY424T2P7EFLBI7QMGQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.806043130 (575h36m35.677126165s) [noise=true , address=45.91.116.13:4153 , piece_id=YN3RPHA2GTMUV3QMERHZZQH477I5L3TEOMV2OEE4N4K33AFANMLQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.835104818 (575h36m35.706187873s) [noise=true , address=45.91.116.14:4189 , piece_id=Z5LVU3A4BDG3N64TQ7EMEDKZDUCNAQERNJHRGNPEAW6PCTJICA4A, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.840790565 (575h36m35.7118736s) [noise=true , address=45.91.116.9:4263 , piece_id=HHH7Y3T63A7X3CBEMGOE6RRGAT3NEC3MUGWOUHIXJSSNQJV6WXNQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.958653292 (575h36m35.829736327s) [noise=true , address=77.90.31.15:4697 , piece_id=FWJAR6DWRS547CN6JM2ZJQI3IYMAMQXFEEA62IYZGF3IF7IF5IYQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.958672539 (575h36m35.829755575s) [seg_pos={0 0}, upload=504590]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:05:35.958873222 (575h36m35.829956257s) [noise=true , address=45.91.116.6:4973 , piece_id=KMVPCXA5TU3G6MGD2E3JBTE6J25WRALAFPP7ZQERKDKWX3MWVOYQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 29 15:05:35.958879944 (575h36m35.82996299s) [noise=true , address=45.91.116.6:4973 , piece_id=KMVPCXA5TU3G6MGD2E3JBTE6J25WRALAFPP7ZQERKDKWX3MWVOYQ, seg_pos={0 0}, upload=504590]: Long tail cancellation due to optimal threshold
Dec 29 15:05:35.958980376 (575h36m35.830063421s) [noise=true , address=145.79.227.35:4483 , piece_id=MSHSUZCFEVIKOSNJCSK2NZTZHTKWIKAHQZDZQP54XXE5DLPTWHXA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 15:05:35.958997648 (575h36m35.830080693s) [noise=true , address=145.79.227.35:4483 , piece_id=MSHSUZCFEVIKOSNJCSK2NZTZHTKWIKAHQZDZQP54XXE5DLPTWHXA, seg_pos={0 0}, upload=504590]: Long tail cancellation due to optimal threshold
Dec 29 15:05:35.959026313 (575h36m35.830109348s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOMPU2T5XSFDDCVDKWVMI6J4HCSV6IT5TVVNN5FEE4C4KQRWNS3Q, seg_pos={0 0}, upload=504590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 29 15:05:35.959034599 (575h36m35.830117634s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOMPU2T5XSFDDCVDKWVMI6J4HCSV6IT5TVVNN5FEE4C4KQRWNS3Q, seg_pos={0 0}, upload=504590]: Long tail cancellation due to optimal threshold
Dec 29 15:05:35.959063814 (575h36m35.830146859s) [seg_pos={0 0}, upload=504590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:05:35.959162613 (575h36m35.830245648s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:05:35.959168103 (575h36m35.830251138s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:05:36.294480588 (575h36m36.165563623s) [upload=504590]: Done uploading segments...
Dec 29 15:05:36.294500126 (575h36m36.165583161s) [upload=504590]: Upload finished. err: <nil>
Duration: 6.209265396s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXX.] duration:
3.482674828s trace:
https://jaeger.dev.storj.io/trace/f47cff950f25740
FAIL (2025-12-29 15:06:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.482703863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XX...XX.X.XXXXX.X..X.....X....X....X...XX.............X....] duration:
29.849552671s trace:
https://jaeger.dev.storj.io/trace/2497001198689567
FAIL (2025-12-29 15:06:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.849583981s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration:
5.001086881s trace:
https://jaeger.dev.storj.io/trace/f47cff950f25740
FAIL (2025-12-29 15:06:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001114743s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X...X] duration:
30.003181772s trace:
https://jaeger.dev.storj.io/trace/2df80b1c0a7a2f74
FAIL (2025-12-29 15:06:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: HHH7Y3T63A7X3CBEMGOE6RRGAT3NEC3MUGWOUHIXJSSNQJV6WXNQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: Z5LVU3A4BDG3N64TQ7EMEDKZDUCNAQERNJHRGNPEAW6PCTJICA4A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: YN3RPHA2GTMUV3QMERHZZQH477I5L3TEOMV2OEE4N4K33AFANMLQ) context deadline exceeded
======LOGS======
Dec 29 15:05:33.320756751 (575h36m33.191839796s) [upload=504590]: Starting upload
Dec 29 15:05:33.320843436 (575h36m33.191926471s) [upload=504590]: Done starting upload
Dec 29 15:05:33.321015714 (575h36m33.192098749s) [upload=504590]: Uploading segments...
Dec 29 15:05:33.321109022 (575h36m33.192192067s) [upload=504590]: Got next segment. Inline: false
Dec 29 15:05:33.321253677 (575h36m33.192336722s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:05:33.321356844 (575h36m33.192439879s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:05:33.571619792 (575h36m33.442702837s) [seg_pos={0 0}, upload=504590]: Begin upload segment...
Dec 29 15:05:33.571700706 (575h36m33.442783751s) [seg_pos={0 0}, upload=504590]: Done begin upload segment.
Dec 29 15:05:33.571735212 (575h36m33.442818257s) [noise=true , address=145.79.227.35:4483 , piece_id=MSHSUZCFEVIKOSNJCSK2NZTZHTKWIKAHQZDZQP54XXE5DLPTWHXA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.571763375 (575h36m33.442846421s) [noise=true , address=45.91.116.13:4153 , piece_id=YN3RPHA2GTMUV3QMERHZZQH477I5L3TEOMV2OEE4N4K33AFANMLQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.571823270 (575h36m33.442906305s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOMPU2T5XSFDDCVDKWVMI6J4HCSV6IT5TVVNN5FEE4C4KQRWNS3Q, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.572381743 (575h36m33.443464778s) [noise=true , address=45.91.116.9:4263 , piece_id=HHH7Y3T63A7X3CBEMGOE6RRGAT3NEC3MUGWOUHIXJSSNQJV6WXNQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.572889009 (575h36m33.443972054s) [noise=true , address=45.91.116.6:4973 , piece_id=KMVPCXA5TU3G6MGD2E3JBTE6J25WRALAFPP7ZQERKDKWX3MWVOYQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.575125186 (575h36m33.446208231s) [noise=true , address=45.91.116.14:4189 , piece_id=Z5LVU3A4BDG3N64TQ7EMEDKZDUCNAQERNJHRGNPEAW6PCTJICA4A, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.575948034 (575h36m33.447031069s) [noise=true , address=145.79.227.36:4517 , piece_id=LEFPRX4ZSDPOWPVKH5ZKOHPPKU72I2XDT3QFINXGQLKIIZQ5SELQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.576999195 (575h36m33.44808223s) [noise=true , address=77.90.31.15:4697 , piece_id=FWJAR6DWRS547CN6JM2ZJQI3IYMAMQXFEEA62IYZGF3IF7IF5IYQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.579632980 (575h36m33.450716016s) [noise=true , address=145.79.227.51:4410 , piece_id=7YL6EUDMHYJDE7TXHRGULLOSSLZF2NK2JW7Q5ZRCOCB5OYGMPHAA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.582358440 (575h36m33.453441485s) [noise=true , address=145.79.227.15:4874 , piece_id=STCGRECFZ4A7IC2MIIKHOJCXJ7ID4BDLHPY424T2P7EFLBI7QMGQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.583788403 (575h36m33.454871438s) [noise=true , address=145.79.227.28:4726 , piece_id=PDEIQUPWNI5SNB74TQDCV5P3IR7Y746HGI4WPPRL67LYMCJ2L2JA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.585498619 (575h36m33.456581664s) [noise=true , address=145.79.227.14:4838 , piece_id=V5GBHM3ZY2A645LQVMB2KVBTSNPX6EHEENNLIUE3VUJPRB7BDACA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.585763243 (575h36m33.456846279s) [noise=true , address=145.79.227.52:4447 , piece_id=EWXB6GEKWJMPCH3S4P6J3XWY7BB4ZV2L7P7ZTH3OCWCC5HMPSC2Q, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.587052338 (575h36m33.458135383s) [noise=true , address=145.79.227.6:4257 , piece_id=6U6WMIJTCIXG4WCBYHWCENZFNXBOD2IE2PPZK5U6BU7IEY6LAFKQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.587650046 (575h36m33.458733091s) [noise=true , address=145.79.227.17:4971 , piece_id=KW72WRRTAABLCIHKNETHTGHSOPQXPAJM2N7VVTUBVGCXG6P6BQPQ, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:33.588353686 (575h36m33.459436731s) [noise=true , address=145.79.227.7:4295 , piece_id=FIORUD3W4D3ZEY7H7CDKIZKQTBV2LKWXMQGTC4TCASAPHNCHTTAA, seg_pos={0 0}, upload=504590]: Uploading piece...
Dec 29 15:05:35.280113881 (575h36m35.151196926s) [upload=504590]: Next returned nil segment
Dec 29 15:05:35.280149249 (575h36m35.151232304s) [upload=504590]: Waiting for error group managing segments...
Dec 29 15:05:35.563124777 (575h36m35.434207812s) [noise=true , address=145.79.227.28:4726 , piece_id=PDEIQUPWNI5SNB74TQDCV5P3IR7Y746HGI4WPPRL67LYMCJ2L2JA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.566350390 (575h36m35.437433425s) [noise=true , address=145.79.227.51:4410 , piece_id=7YL6EUDMHYJDE7TXHRGULLOSSLZF2NK2JW7Q5ZRCOCB5OYGMPHAA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.567063728 (575h36m35.438146763s) [noise=true , address=145.79.227.6:4257 , piece_id=6U6WMIJTCIXG4WCBYHWCENZFNXBOD2IE2PPZK5U6BU7IEY6LAFKQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.569454370 (575h36m35.440537405s) [noise=true , address=145.79.227.14:4838 , piece_id=V5GBHM3ZY2A645LQVMB2KVBTSNPX6EHEENNLIUE3VUJPRB7BDACA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.574928824 (575h36m35.446011869s) [noise=true , address=145.79.227.17:4971 , piece_id=KW72WRRTAABLCIHKNETHTGHSOPQXPAJM2N7VVTUBVGCXG6P6BQPQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.576093883 (575h36m35.447176918s) [noise=true , address=145.79.227.36:4517 , piece_id=LEFPRX4ZSDPOWPVKH5ZKOHPPKU72I2XDT3QFINXGQLKIIZQ5SELQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.580933889 (575h36m35.452016934s) [noise=true , address=145.79.227.52:4447 , piece_id=EWXB6GEKWJMPCH3S4P6J3XWY7BB4ZV2L7P7ZTH3OCWCC5HMPSC2Q, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.594457560 (575h36m35.465540605s) [noise=true , address=145.79.227.7:4295 , piece_id=FIORUD3W4D3ZEY7H7CDKIZKQTBV2LKWXMQGTC4TCASAPHNCHTTAA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.671937163 (575h36m35.543020208s) [noise=true , address=145.79.227.15:4874 , piece_id=STCGRECFZ4A7IC2MIIKHOJCXJ7ID4BDLHPY424T2P7EFLBI7QMGQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.806043130 (575h36m35.677126165s) [noise=true , address=45.91.116.13:4153 , piece_id=YN3RPHA2GTMUV3QMERHZZQH477I5L3TEOMV2OEE4N4K33AFANMLQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.835104818 (575h36m35.706187873s) [noise=true , address=45.91.116.14:4189 , piece_id=Z5LVU3A4BDG3N64TQ7EMEDKZDUCNAQERNJHRGNPEAW6PCTJICA4A, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.840790565 (575h36m35.7118736s) [noise=true , address=45.91.116.9:4263 , piece_id=HHH7Y3T63A7X3CBEMGOE6RRGAT3NEC3MUGWOUHIXJSSNQJV6WXNQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.958653292 (575h36m35.829736327s) [noise=true , address=77.90.31.15:4697 , piece_id=FWJAR6DWRS547CN6JM2ZJQI3IYMAMQXFEEA62IYZGF3IF7IF5IYQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: <nil>
Dec 29 15:05:35.958672539 (575h36m35.829755575s) [seg_pos={0 0}, upload=504590]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:05:35.958873222 (575h36m35.829956257s) [noise=true , address=45.91.116.6:4973 , piece_id=KMVPCXA5TU3G6MGD2E3JBTE6J25WRALAFPP7ZQERKDKWX3MWVOYQ, seg_pos={0 0}, upload=504590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 29 15:05:35.958879944 (575h36m35.82996299s) [noise=true , address=45.91.116.6:4973 , piece_id=KMVPCXA5TU3G6MGD2E3JBTE6J25WRALAFPP7ZQERKDKWX3MWVOYQ, seg_pos={0 0}, upload=504590]: Long tail cancellation due to optimal threshold
Dec 29 15:05:35.958980376 (575h36m35.830063421s) [noise=true , address=145.79.227.35:4483 , piece_id=MSHSUZCFEVIKOSNJCSK2NZTZHTKWIKAHQZDZQP54XXE5DLPTWHXA, seg_pos={0 0}, upload=504590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 15:05:35.958997648 (575h36m35.830080693s) [noise=true , address=145.79.227.35:4483 , piece_id=MSHSUZCFEVIKOSNJCSK2NZTZHTKWIKAHQZDZQP54XXE5DLPTWHXA, seg_pos={0 0}, upload=504590]: Long tail cancellation due to optimal threshold
Dec 29 15:05:35.959026313 (575h36m35.830109348s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOMPU2T5XSFDDCVDKWVMI6J4HCSV6IT5TVVNN5FEE4C4KQRWNS3Q, seg_pos={0 0}, upload=504590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 29 15:05:35.959034599 (575h36m35.830117634s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOMPU2T5XSFDDCVDKWVMI6J4HCSV6IT5TVVNN5FEE4C4KQRWNS3Q, seg_pos={0 0}, upload=504590]: Long tail cancellation due to optimal threshold
Dec 29 15:05:35.959063814 (575h36m35.830146859s) [seg_pos={0 0}, upload=504590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:05:35.959162613 (575h36m35.830245648s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:05:35.959168103 (575h36m35.830251138s) [upload=504590]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:05:36.294480588 (575h36m36.165563623s) [upload=504590]: Done uploading segments...
Dec 29 15:05:36.294500126 (575h36m36.165583161s) [upload=504590]: Upload finished. err: <nil>
Duration: 30.003205797s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [....................XXX...........XX.X......................] duration:
6.477658224s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:06:02 UTC)
orbiter warning: context deadline exceeded
Duration: 6.477686599s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXX] duration:
5.001523309s trace:
https://jaeger.dev.storj.io/trace/496b293b85db5266
FAIL (2025-12-29 15:05:50 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001546423s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............X........X.........X......X.......X...X.......] duration:
51.683642993s trace:
https://jaeger.dev.storj.io/trace/24cdf967807c73aa
FAIL (2025-12-29 15:05:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded
======LOGS======
575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment
Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments...
Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection
Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments...
Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil>
Duration: 51.683684182s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
7.923736695s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:05:40 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 7.923926296s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.....................XXX...........XX.X.....................] duration:
36.331723248s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:05:38 UTC)
orbiter warning: context deadline exceeded
Duration: 36.331788682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..XX...XX.X.XXXXX.X..X.....X....X....X...XX.............X...] duration:
23.962733917s trace:
https://jaeger.dev.storj.io/trace/2497001198689567
FAIL (2025-12-29 15:05:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.962759897s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XX] duration:
5.001071218s trace:
https://jaeger.dev.storj.io/trace/64206b6f200a47dd
FAIL (2025-12-29 15:05:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001113589s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.X] duration:
5.001289653s trace:
https://jaeger.dev.storj.io/trace/2678ffde340b17a4
FAIL (2025-12-29 15:04:57 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001304732s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X...] duration:
5.98397254s trace:
https://jaeger.dev.storj.io/trace/702e1a3c40e071fa
FAIL (2025-12-29 15:04:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 15:02:50.701227468 (575h33m50.572310503s) [upload=504556]: Starting upload
Dec 29 15:02:50.701363046 (575h33m50.572446091s) [upload=504556]: Done starting upload
Dec 29 15:02:50.701618793 (575h33m50.572701838s) [upload=504556]: Uploading segments...
Dec 29 15:02:50.701668668 (575h33m50.572751714s) [upload=504556]: Got next segment. Inline: false
Dec 29 15:02:50.701701010 (575h33m50.572784045s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:02:50.701706220 (575h33m50.572789255s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:02:50.947509747 (575h33m50.818592792s) [seg_pos={0 0}, upload=504556]: Begin upload segment...
Dec 29 15:02:50.947563980 (575h33m50.818647015s) [seg_pos={0 0}, upload=504556]: Done begin upload segment.
Dec 29 15:02:50.947597634 (575h33m50.818680679s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947640305 (575h33m50.81872334s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947744273 (575h33m50.818827318s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948295112 (575h33m50.819378157s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948724810 (575h33m50.819807855s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949131945 (575h33m50.820214991s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949644953 (575h33m50.820727998s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950078087 (575h33m50.821161122s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950504049 (575h33m50.821587084s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.952658110 (575h33m50.823741155s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.954399415 (575h33m50.82548246s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.956064576 (575h33m50.827147611s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.957699199 (575h33m50.828782234s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.959484719 (575h33m50.830567764s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.961283905 (575h33m50.83236695s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.962191904 (575h33m50.833274949s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:52.646677396 (575h33m52.517760441s) [upload=504556]: Next returned nil segment
Dec 29 15:02:52.646693147 (575h33m52.517776182s) [upload=504556]: Waiting for error group managing segments...
Dec 29 15:02:52.929771059 (575h33m52.800854094s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.941900515 (575h33m52.81298356s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962515018 (575h33m52.833598054s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962797476 (575h33m52.833880521s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.971366052 (575h33m52.842449107s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974205118 (575h33m52.845288163s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974942312 (575h33m52.846025347s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.172408857 (575h33m53.043491902s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.230215170 (575h33m53.101298215s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.247996648 (575h33m53.119079693s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.269960580 (575h33m53.141043625s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.333047026 (575h33m53.204130061s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337777604 (575h33m53.208860639s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337823802 (575h33m53.208906847s) [seg_pos={0 0}, upload=504556]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:02:53.338100459 (575h33m53.209183504s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338109606 (575h33m53.209192641s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338210368 (575h33m53.209293403s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338225267 (575h33m53.209308302s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338376965 (575h33m53.20946s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338383087 (575h33m53.209466132s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338416771 (575h33m53.209499816s) [seg_pos={0 0}, upload=504556]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:02:53.338518154 (575h33m53.209601199s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:02:53.338527612 (575h33m53.209610647s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:02:53.626254487 (575h33m53.497337542s) [upload=504556]: Done uploading segments...
Dec 29 15:02:53.626281308 (575h33m53.497364353s) [upload=504556]: Upload finished. err: <nil>
Duration: 5.984004581s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [...XX...XX.X.XXXXX.X..X.....X....X....X...XX.............X..] duration:
16.633619356s trace:
https://jaeger.dev.storj.io/trace/2497001198689567
FAIL (2025-12-29 15:04:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.633646077s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [......................XXX...........XX.X....................] duration:
15.136872013s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:04:53 UTC)
orbiter warning: context deadline exceeded
Duration: 15.136901188s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
9.89978529s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:04:46 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 9.899808534s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............X........X.........X......X.......X...X......] duration:
41.149990148s trace:
https://jaeger.dev.storj.io/trace/24cdf967807c73aa
FAIL (2025-12-29 15:04:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded
======LOGS======
575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment
Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments...
Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection
Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments...
Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil>
Duration: 41.150018533s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.............X.] duration:
12.562031057s trace:
https://jaeger.dev.storj.io/trace/2497001198689567
FAIL (2025-12-29 15:04:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.562059522s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X..] duration:
6.07056139s trace:
https://jaeger.dev.storj.io/trace/702e1a3c40e071fa
FAIL (2025-12-29 15:04:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 15:02:50.701227468 (575h33m50.572310503s) [upload=504556]: Starting upload
Dec 29 15:02:50.701363046 (575h33m50.572446091s) [upload=504556]: Done starting upload
Dec 29 15:02:50.701618793 (575h33m50.572701838s) [upload=504556]: Uploading segments...
Dec 29 15:02:50.701668668 (575h33m50.572751714s) [upload=504556]: Got next segment. Inline: false
Dec 29 15:02:50.701701010 (575h33m50.572784045s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:02:50.701706220 (575h33m50.572789255s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:02:50.947509747 (575h33m50.818592792s) [seg_pos={0 0}, upload=504556]: Begin upload segment...
Dec 29 15:02:50.947563980 (575h33m50.818647015s) [seg_pos={0 0}, upload=504556]: Done begin upload segment.
Dec 29 15:02:50.947597634 (575h33m50.818680679s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947640305 (575h33m50.81872334s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947744273 (575h33m50.818827318s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948295112 (575h33m50.819378157s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948724810 (575h33m50.819807855s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949131945 (575h33m50.820214991s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949644953 (575h33m50.820727998s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950078087 (575h33m50.821161122s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950504049 (575h33m50.821587084s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.952658110 (575h33m50.823741155s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.954399415 (575h33m50.82548246s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.956064576 (575h33m50.827147611s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.957699199 (575h33m50.828782234s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.959484719 (575h33m50.830567764s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.961283905 (575h33m50.83236695s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.962191904 (575h33m50.833274949s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:52.646677396 (575h33m52.517760441s) [upload=504556]: Next returned nil segment
Dec 29 15:02:52.646693147 (575h33m52.517776182s) [upload=504556]: Waiting for error group managing segments...
Dec 29 15:02:52.929771059 (575h33m52.800854094s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.941900515 (575h33m52.81298356s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962515018 (575h33m52.833598054s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962797476 (575h33m52.833880521s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.971366052 (575h33m52.842449107s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974205118 (575h33m52.845288163s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974942312 (575h33m52.846025347s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.172408857 (575h33m53.043491902s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.230215170 (575h33m53.101298215s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.247996648 (575h33m53.119079693s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.269960580 (575h33m53.141043625s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.333047026 (575h33m53.204130061s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337777604 (575h33m53.208860639s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337823802 (575h33m53.208906847s) [seg_pos={0 0}, upload=504556]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:02:53.338100459 (575h33m53.209183504s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338109606 (575h33m53.209192641s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338210368 (575h33m53.209293403s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338225267 (575h33m53.209308302s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338376965 (575h33m53.20946s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338383087 (575h33m53.209466132s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338416771 (575h33m53.209499816s) [seg_pos={0 0}, upload=504556]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:02:53.338518154 (575h33m53.209601199s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:02:53.338527612 (575h33m53.209610647s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:02:53.626254487 (575h33m53.497337542s) [upload=504556]: Done uploading segments...
Dec 29 15:02:53.626281308 (575h33m53.497364353s) [upload=504556]: Upload finished. err: <nil>
Duration: 6.070590165s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X.] duration:
7.0751133s trace:
https://jaeger.dev.storj.io/trace/702e1a3c40e071fa
FAIL (2025-12-29 15:03:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 15:02:50.701227468 (575h33m50.572310503s) [upload=504556]: Starting upload
Dec 29 15:02:50.701363046 (575h33m50.572446091s) [upload=504556]: Done starting upload
Dec 29 15:02:50.701618793 (575h33m50.572701838s) [upload=504556]: Uploading segments...
Dec 29 15:02:50.701668668 (575h33m50.572751714s) [upload=504556]: Got next segment. Inline: false
Dec 29 15:02:50.701701010 (575h33m50.572784045s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:02:50.701706220 (575h33m50.572789255s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:02:50.947509747 (575h33m50.818592792s) [seg_pos={0 0}, upload=504556]: Begin upload segment...
Dec 29 15:02:50.947563980 (575h33m50.818647015s) [seg_pos={0 0}, upload=504556]: Done begin upload segment.
Dec 29 15:02:50.947597634 (575h33m50.818680679s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947640305 (575h33m50.81872334s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947744273 (575h33m50.818827318s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948295112 (575h33m50.819378157s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948724810 (575h33m50.819807855s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949131945 (575h33m50.820214991s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949644953 (575h33m50.820727998s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950078087 (575h33m50.821161122s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950504049 (575h33m50.821587084s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.952658110 (575h33m50.823741155s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.954399415 (575h33m50.82548246s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.956064576 (575h33m50.827147611s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.957699199 (575h33m50.828782234s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.959484719 (575h33m50.830567764s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.961283905 (575h33m50.83236695s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.962191904 (575h33m50.833274949s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:52.646677396 (575h33m52.517760441s) [upload=504556]: Next returned nil segment
Dec 29 15:02:52.646693147 (575h33m52.517776182s) [upload=504556]: Waiting for error group managing segments...
Dec 29 15:02:52.929771059 (575h33m52.800854094s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.941900515 (575h33m52.81298356s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962515018 (575h33m52.833598054s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962797476 (575h33m52.833880521s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.971366052 (575h33m52.842449107s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974205118 (575h33m52.845288163s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974942312 (575h33m52.846025347s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.172408857 (575h33m53.043491902s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.230215170 (575h33m53.101298215s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.247996648 (575h33m53.119079693s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.269960580 (575h33m53.141043625s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.333047026 (575h33m53.204130061s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337777604 (575h33m53.208860639s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337823802 (575h33m53.208906847s) [seg_pos={0 0}, upload=504556]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:02:53.338100459 (575h33m53.209183504s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338109606 (575h33m53.209192641s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338210368 (575h33m53.209293403s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338225267 (575h33m53.209308302s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338376965 (575h33m53.20946s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338383087 (575h33m53.209466132s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338416771 (575h33m53.209499816s) [seg_pos={0 0}, upload=504556]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:02:53.338518154 (575h33m53.209601199s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:02:53.338527612 (575h33m53.209610647s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:02:53.626254487 (575h33m53.497337542s) [upload=504556]: Done uploading segments...
Dec 29 15:02:53.626281308 (575h33m53.497364353s) [upload=504556]: Upload finished. err: <nil>
Duration: 7.075134651s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.......................XXX...........XX.X...................] duration:
14.841783075s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:03:49 UTC)
orbiter warning: context deadline exceeded
Duration: 14.841804385s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.096915192s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:03:49 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.096941213s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [................X........X.........X......X.......X...X.....] duration:
15.522500432s trace:
https://jaeger.dev.storj.io/trace/24cdf967807c73aa
FAIL (2025-12-29 15:03:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded
======LOGS======
575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment
Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments...
Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection
Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments...
Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil>
Duration: 15.522518425s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.] duration:
3.762412039s trace:
https://jaeger.dev.storj.io/trace/3dd429f8e224f764
FAIL (2025-12-29 15:03:44 UTC)
orbiter warning: context deadline exceeded
Duration: 3.762434853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.............X] duration:
30.001525907s trace:
https://jaeger.dev.storj.io/trace/2497001198689567
FAIL (2025-12-29 15:03:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001576092s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
4.728743267s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:03:31 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 4.728814874s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....X.X] duration:
30.003335734s trace:
https://jaeger.dev.storj.io/trace/702e1a3c40e071fa
FAIL (2025-12-29 15:03:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 15:02:50.701227468 (575h33m50.572310503s) [upload=504556]: Starting upload
Dec 29 15:02:50.701363046 (575h33m50.572446091s) [upload=504556]: Done starting upload
Dec 29 15:02:50.701618793 (575h33m50.572701838s) [upload=504556]: Uploading segments...
Dec 29 15:02:50.701668668 (575h33m50.572751714s) [upload=504556]: Got next segment. Inline: false
Dec 29 15:02:50.701701010 (575h33m50.572784045s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:02:50.701706220 (575h33m50.572789255s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:02:50.947509747 (575h33m50.818592792s) [seg_pos={0 0}, upload=504556]: Begin upload segment...
Dec 29 15:02:50.947563980 (575h33m50.818647015s) [seg_pos={0 0}, upload=504556]: Done begin upload segment.
Dec 29 15:02:50.947597634 (575h33m50.818680679s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947640305 (575h33m50.81872334s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.947744273 (575h33m50.818827318s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948295112 (575h33m50.819378157s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.948724810 (575h33m50.819807855s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949131945 (575h33m50.820214991s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.949644953 (575h33m50.820727998s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950078087 (575h33m50.821161122s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.950504049 (575h33m50.821587084s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.952658110 (575h33m50.823741155s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.954399415 (575h33m50.82548246s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.956064576 (575h33m50.827147611s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.957699199 (575h33m50.828782234s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.959484719 (575h33m50.830567764s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.961283905 (575h33m50.83236695s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:50.962191904 (575h33m50.833274949s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Uploading piece...
Dec 29 15:02:52.646677396 (575h33m52.517760441s) [upload=504556]: Next returned nil segment
Dec 29 15:02:52.646693147 (575h33m52.517776182s) [upload=504556]: Waiting for error group managing segments...
Dec 29 15:02:52.929771059 (575h33m52.800854094s) [noise=true , address=145.79.227.14:4840 , piece_id=57PW24KERH2ZZ3LMHLREZISVRJ2SBW4RM43HLFMLIWL3ZPYZZKMA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.941900515 (575h33m52.81298356s) [noise=true , address=145.79.227.39:4013 , piece_id=EWSEX2BRLW673YESCVC5FCUOCZEENPQ5FX6QL2KBN4JVV4KWYTHQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962515018 (575h33m52.833598054s) [noise=true , address=145.79.227.37:4566 , piece_id=KD4XUONWPBEHQTMTAMCTHKEVG5OYA2ER3ARIVLF6PEVDZXE7KUSA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.962797476 (575h33m52.833880521s) [noise=true , address=145.79.227.12:4767 , piece_id=HF6GYFQGF4XXBERABYGK7U2AKFP5EMQCOJSQV6OASX63IQ2VDRHA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.971366052 (575h33m52.842449107s) [noise=true , address=145.79.227.47:4267 , piece_id=RWQO6O462XCRU4D6RGM6FWNHCHHF3IITI3W4NZRG3LCOS3X3KPJA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974205118 (575h33m52.845288163s) [noise=true , address=145.79.227.10:4403 , piece_id=5PBN45LMVZOQF3NH3I7TQ7Y7IB7F6SW332IRY5ABIYC7V62NL6IA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:52.974942312 (575h33m52.846025347s) [noise=true , address=145.79.227.19:4079 , piece_id=7CXEUGRF4WUZRJ5MSJ66NWSSXW2RFSXDYDDCBYDCZBVDCFET2IOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.172408857 (575h33m53.043491902s) [noise=true , address=45.91.116.7:4012 , piece_id=HQG4TIS6K3JLQSAF536TOJCD5K7FEEV4TXRG4PUOLVMW3BCVWLTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.230215170 (575h33m53.101298215s) [noise=true , address=45.91.116.12:4226 , piece_id=4JS3OFJGOJSMON2TETSED3WWW5UEUBBD4XW6LRAORCVH3JSINJLQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.247996648 (575h33m53.119079693s) [noise=true , address=77.90.31.15:4697 , piece_id=22KGLGWBJQNSYTAXNFKAFL7PA23LFQOXEGKQVHCFVDZCHAA3G7FA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.269960580 (575h33m53.141043625s) [noise=true , address=77.90.31.16:4736 , piece_id=AZ4HSWZ2EVTCMF2R67DRYCMMI3CDR6FXQ2JWJ7DY3GZXFOCU7XSQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.333047026 (575h33m53.204130061s) [noise=true , address=45.91.116.14:4189 , piece_id=SYSQRGTFCFJOBXC6ESYV7DIRPAS4AUZM5JXYU6HG3SKB3T5NDMCQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337777604 (575h33m53.208860639s) [noise=true , address=45.91.116.11:4117 , piece_id=M5W4X3ZYRWJ52R6HNZDVT6LXRVSQXH4IEZQRHGIL36YLPPAUQILA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: <nil>
Dec 29 15:02:53.337823802 (575h33m53.208906847s) [seg_pos={0 0}, upload=504556]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:02:53.338100459 (575h33m53.209183504s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338109606 (575h33m53.209192641s) [noise=true , address=145.79.227.35:4482 , piece_id=ZH7GUAJJ4QSYDEU6BQDKCY2F7ICVHJXXGJ4FUI3EOSZUF4PRUJTA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338210368 (575h33m53.209293403s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338225267 (575h33m53.209308302s) [noise=true , address=77.90.31.11:4556 , piece_id=HREGV6RLFGH3KHIP7MFCPRTWBQGGKPX5CJYFKT3Y7KYZ5DRHWCFA, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338376965 (575h33m53.20946s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 29 15:02:53.338383087 (575h33m53.209466132s) [noise=true , address=145.79.227.29:4941 , piece_id=GRA3BSHYJBK5DND26Z54MXTK6X5DGFDEACF2DAKLQAFHRPCKUDOQ, seg_pos={0 0}, upload=504556]: Long tail cancellation due to optimal threshold
Dec 29 15:02:53.338416771 (575h33m53.209499816s) [seg_pos={0 0}, upload=504556]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:02:53.338518154 (575h33m53.209601199s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:02:53.338527612 (575h33m53.209610647s) [upload=504556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:02:53.626254487 (575h33m53.497337542s) [upload=504556]: Done uploading segments...
Dec 29 15:02:53.626281308 (575h33m53.497364353s) [upload=504556]: Upload finished. err: <nil>
Duration: 30.003361002s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
8.131385098s trace:
https://jaeger.dev.storj.io/trace/6d06116a481c3bb9
FAIL (2025-12-29 15:03:13 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.131411719s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................X........X.........X......X.......X...X....] duration:
50.730141645s trace:
https://jaeger.dev.storj.io/trace/24cdf967807c73aa
FAIL (2025-12-29 15:03:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded
======LOGS======
575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece...
Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment
Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments...
Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil>
Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection
Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold
Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments...
Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil>
Duration: 50.730167063s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [........................XXX...........XX.X..................] duration:
16.229793381s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:03:06 UTC)
orbiter warning: context deadline exceeded
Duration: 16.229809021s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.............] duration:
15.755611377s trace:
https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75
FAIL (2025-12-29 15:02:53 UTC)
orbiter warning: context deadline exceeded
Duration: 15.755637327s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........................XXX...........XX.X.................] duration:
41.687917806s trace:
https://jaeger.dev.storj.io/trace/601575448e35e827
FAIL (2025-12-29 15:02:50 UTC)
orbiter warning: context deadline exceeded
Duration: 41.687943807s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001563046s trace:
https://jaeger.dev.storj.io/trace/3dd429f8e224f764
FAIL (2025-12-29 15:02:43 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001582994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX............] duration:
29.581360668s trace:
https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75
FAIL (2025-12-29 15:02:17 UTC)
orbiter warning: context deadline exceeded
Duration: 29.58138785s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration:
5.000741945s trace:
https://jaeger.dev.storj.io/trace/56bf4a2253997ff9
FAIL (2025-12-29 15:02:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000768174s
core
uplink
us-select-1
us1-placement-20
|