Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001611582s
FAIL (2025-10-07 10:39:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001735539s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.851902885s
FAIL (2025-10-07 10:38:42 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.851929926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.304376492s
FAIL (2025-10-07 10:38:27 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.304424043s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
2.149174491s
FAIL (2025-10-07 10:38:23 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.149201891s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.749420452s
FAIL (2025-10-07 10:38:04 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.749453074s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.176776119s
FAIL (2025-10-07 10:38:01 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.176889395s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
2.106961249s
FAIL (2025-10-07 10:37:53 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.106995423s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.166038924s
FAIL (2025-10-07 10:37:45 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.166092095s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
5.775934699s
FAIL (2025-10-07 10:37:34 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.775966929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.129414372s
FAIL (2025-10-07 10:37:28 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.129451815s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.878395802s
FAIL (2025-10-07 10:36:35 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 5.878429307s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.855894546s
FAIL (2025-10-07 10:36:35 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.855937578s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
2.198096756s
FAIL (2025-10-07 10:36:17 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.198130511s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.050011917s
FAIL (2025-10-07 10:35:58 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.050042436s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.017796555s
FAIL (2025-10-07 10:35:54 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 6.017847173s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
1.963270988s
FAIL (2025-10-07 10:35:42 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 1.963309431s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
6.05794222s
FAIL (2025-10-07 10:35:40 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.05796909s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
6.100607501s
FAIL (2025-10-07 10:35:19 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.100645384s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.029833624s
FAIL (2025-10-07 10:35:06 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.029866115s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.951584018s
FAIL (2025-10-07 10:34:43 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.95161677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.050176042s
FAIL (2025-10-07 10:34:36 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.050214935s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.024945354s
FAIL (2025-10-07 10:34:19 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.024975533s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.187408278s
FAIL (2025-10-07 10:34:10 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.187442933s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
6.159917034s
FAIL (2025-10-07 10:33:36 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 6.159943473s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.200824914s
FAIL (2025-10-07 10:33:31 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.200855443s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
6.277799512s
FAIL (2025-10-07 10:33:17 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.277828978s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
2.154076129s
FAIL (2025-10-07 10:33:12 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.15410897s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
5.73522331s
FAIL (2025-10-07 10:32:58 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.735258236s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
6.125192792s
FAIL (2025-10-07 10:32:52 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.125224744s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.84640246s
FAIL (2025-10-07 10:32:34 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.846433258s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.145305254s
FAIL (2025-10-07 10:32:22 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.145344409s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.030151837s
FAIL (2025-10-07 10:32:18 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.030258481s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.006665436s
FAIL (2025-10-07 10:31:54 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.006699291s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
2.104193431s
FAIL (2025-10-07 10:31:35 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.10422529s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.649627939s
FAIL (2025-10-07 10:31:30 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.649666023s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
6.202665873s
FAIL (2025-10-07 10:31:16 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.202701821s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
2.204994876s
FAIL (2025-10-07 10:31:16 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.205032689s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.941453555s
FAIL (2025-10-07 10:30:22 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.941484886s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.074337874s
FAIL (2025-10-07 10:30:14 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.074371818s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.054907289s
FAIL (2025-10-07 10:30:11 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.054946986s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.069596038s
FAIL (2025-10-07 10:29:45 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.069629941s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.265363069s
FAIL (2025-10-07 10:29:22 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.265398718s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.940104457s
FAIL (2025-10-07 10:29:08 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.940135106s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.138527933s
FAIL (2025-10-07 10:29:03 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.138561577s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.088491791s
FAIL (2025-10-07 10:28:54 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.088524413s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
5.970389153s
FAIL (2025-10-07 10:28:45 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.970417315s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
5.944445613s
FAIL (2025-10-07 10:28:14 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2:42.165264348 (332h41m27.627356861s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.165955152 (332h41m27.628047625s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.166898680 (332h41m27.628991163s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167359617 (332h41m27.62945209s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.167858417 (332h41m27.6299509s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.168663159 (332h41m27.630755642s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169008787 (332h41m27.63110126s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169409100 (332h41m27.631501583s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.169959709 (332h41m27.632052192s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.170441054 (332h41m27.632533537s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171142340 (332h41m27.633234834s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171422123 (332h41m27.633514606s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.171936802 (332h41m27.634029286s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172419642 (332h41m27.634512165s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.172505987 (332h41m27.63459847s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173003063 (332h41m27.635095546s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:42.173354913 (332h41m27.635447396s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Uploading piece...
Oct 7 10:12:43.558596988 (332h41m29.020689461s) [upload=292777]: Next returned nil segment
Oct 7 10:12:43.558619280 (332h41m29.020711753s) [upload=292777]: Waiting for error group managing segments...
Oct 7 10:12:43.911075424 (332h41m29.373167907s) [noise=true , address=64.49.13.27:4959 , piece_id=Q7IVCXASNNEVEJF6PSNQOFMX4GOR4VQNWXI4K67H4T3WJRGRERVA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:43.999387778 (332h41m29.461480251s) [noise=true , address=145.79.227.23:4575 , piece_id=RBUQMX6L7HRE7777AIAMNUEIRHR6AFC33DMVQANBKEGUX5DVXEBA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.008759212 (332h41m29.470851685s) [noise=true , address=145.79.227.42:4138 , piece_id=B2EG7P62T6Z7MV3WAKFRGYH3WQNNHDWYQL5ONKYJRN7BIBCVI2CA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010009934 (332h41m29.472102417s) [noise=true , address=145.79.227.19:4106 , piece_id=5W2IHZIDRVT4N65DW5EKD54QQSQAU5CC4IAENWMQIIMSMMBJIAOQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.010760634 (332h41m29.472853117s) [noise=true , address=145.79.227.24:4591 , piece_id=XUGNOWN33T62WB5L7CPI4BDNIJHHYNWKCW3RFFGVCSX7JQICJNIQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.027043800 (332h41m29.489136283s) [noise=true , address=145.79.227.20:4447 , piece_id=OJMCBKSVDODKQBP63HGNCE2J6CG7ZGJTJHC5ZHY6HO552L3IWVJA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029164559 (332h41m29.491257032s) [noise=true , address=145.79.227.29:4952 , piece_id=N3I32ETGCV5Q7VPWSIH5NBSU3DWHL3B4MVY5AGIL35YUGPIPWVYA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.029210005 (332h41m29.491302488s) [noise=true , address=145.79.227.38:4004 , piece_id=JBNMBUDYP6L4DDUZMLKWC56XR4C7XYVXV7ECBUUTMOIKSHJOZ6AQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.033972373 (332h41m29.496064856s) [noise=true , address=145.79.227.32:4840 , piece_id=2VMNF4ESFOC4FD6XIJXOH2OYW7KOIP5FTWNC43BJOKMRCRM2XP2A, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.034737540 (332h41m29.496830013s) [noise=true , address=145.79.227.52:4458 , piece_id=PWK65JVKZJOKTRJ3RUFWRARU6OMYAQJ45ALDFY47BIHBOWSMA2XA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.142533886 (332h41m29.604626369s) [noise=true , address=77.90.31.15:4730 , piece_id=DNLCZ75DDJ727UKPE4BLQ4RW2F7CONINHHO7RE325D5EWWA5JDTA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.158687164 (332h41m29.620779637s) [noise=true , address=77.90.31.18:4814 , piece_id=VDA7BKMPAMNXMUZXYKLD3ECQRLPTQTFCRCOZSBUXQ34WRUAR6JFA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.174538559 (332h41m29.636631032s) [noise=true , address=45.91.116.5:4312 , piece_id=5P6KE7HIRAEGY2PRHLZSB42LZS6RRBZWUQGGMP3EISRETUT6YR5Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206699178 (332h41m29.668791661s) [noise=true , address=45.91.116.14:4215 , piece_id=45BEDZCAYMQWZJBETXDADIBBG2A6LKYMVXI46HBYVUKR6S6JGJHA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: <nil>
Oct 7 10:12:44.206743001 (332h41m29.668835484s) [seg_pos={0 0}, upload=292777]: Segment reached optimal threshold of 14 pieces.
Oct 7 10:12:44.206951859 (332h41m29.669044342s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.206961288 (332h41m29.669053771s) [noise=true , address=77.90.31.21:4885 , piece_id=QZQ7O2X56Y5Y24LG6OHRHGZMHZJ5ZGB7ACZMCDRPK3U7HSLAUAZA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207002376 (332h41m29.669094859s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207022453 (332h41m29.669114936s) [noise=true , address=45.91.116.8:4064 , piece_id=RB5BLW65DRMAEZ5DZNAOKBTX2CGBIAEJXGREKNJQCEZJEEQYMZ4Q, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207077298 (332h41m29.669169781s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207086465 (332h41m29.669178938s) [noise=true , address=45.91.116.4:4963 , piece_id=D43ZVDFIGGIPIXQMMIHJOSQK6EHS54RTKUYMAK5I4XTB4LNLHKYQ, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207177830 (332h41m29.669270303s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF
Oct 7 10:12:44.207186476 (332h41m29.669278949s) [noise=true , address=77.90.31.19:4860 , piece_id=QRWWQSWR6HLKTJ4R6PREPXYC5QONDP4T6DWNBG5KGT6K6TY2TUPA, seg_pos={0 0}, upload=292777]: Long tail cancellation due to optimal threshold
Oct 7 10:12:44.207212376 (332h41m29.669304859s) [seg_pos={0 0}, upload=292777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 7 10:12:44.207297427 (332h41m29.66938991s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 7 10:12:44.207302326 (332h41m29.669394799s) [upload=292777]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 7 10:12:52.521007048 (332h41m37.983099531s) [upload=292777]: Done uploading segments...
Oct 7 10:12:52.521031375 (332h41m37.983123848s) [upload=292777]: Upload finished. err: <nil>
Duration: 5.944474159s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
6.275755483s
FAIL (2025-10-07 10:28:08 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 6.275791361s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.130672515s
FAIL (2025-10-07 10:27:43 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 2.130705328s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.917571657s
FAIL (2025-10-07 10:27:23 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 7 10:12:08.558107699 (332h40m54.020200182s) [upload=292768]: Starting upload Oct 7 10:12:08.558204974 (332h40m54.020297457s) [upload=292768]: Done starting upload Oct 7 10:12:08.558299444 (332h40m54.020391927s) [upload=292768]: Uploading segments... Oct 7 10:12:08.558492031 (332h40m54.020584514s) [upload=292768]: Got next segment. Inline: false Oct 7 10:12:08.558535513 (332h40m54.020627996s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 7 10:12:08.558544311 (332h40m54.020636804s) [upload=292768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 7 10:12:37.855472289 (332h41m23.317564762s) [upload=292768]: Next segment err: context deadline exceeded Oct 7 10:12:37.855578451 (332h41m23.317670924s) [upload=292768]: Done uploading segments... Oct 7 10:12:37.855594531 (332h41m23.317687014s) [upload=292768]: Upload finished. err: context deadline exceeded
Duration: 5.917634417s
core
uplink
us-select-1
|