Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 342623 reports (Page 163 of 6853)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [X..XXXXX....X...X..X..X...X...X.X.........X.....X......X.X.X]
duration: 
30.002497258s
trace: 
https://jaeger.dev.storj.io/trace/72be81bed72094ce
FAIL (2025-12-28 15:03:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 15:03:08.068822952 (551h34m7.939906007s) [upload=483604]: Starting upload
Dec 28 15:03:08.068936018 (551h34m7.940019063s) [upload=483604]: Done starting upload
Dec 28 15:03:08.069035928 (551h34m7.940118973s) [upload=483604]: Uploading segments...
Dec 28 15:03:08.069278580 (551h34m7.940361615s) [upload=483604]: Got next segment. Inline: false
Dec 28 15:03:08.069320030 (551h34m7.940403075s) [upload=483604]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:03:08.069326852 (551h34m7.940409948s) [upload=483604]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:03:08.320445504 (551h34m8.191528549s) [seg_pos={0 0}, upload=483604]: Begin upload segment...
Dec 28 15:03:08.320488747 (551h34m8.191571792s) [seg_pos={0 0}, upload=483604]: Done begin upload segment.
Dec 28 15:03:08.320512011 (551h34m8.191595046s) [noise=true , address=145.79.227.52:4445   , piece_id=AXI74ZBRV66TBGA6UYDSVGKK3JDW7OGK7N44QC4TSWL7RA6Y33UA, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.320616059 (551h34m8.191699104s) [noise=true , address=145.79.227.51:4411   , piece_id=ZMN6BDN5ILCAVQ7NPEMDTFIA3KU3XRDKDNMVGHDLNAREGPORUQ3Q, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.321450508 (551h34m8.192533553s) [noise=true , address=145.79.227.16:4909   , piece_id=2PNWTZ2LFVNIIL4GH4QNV2BM6O6VBZUEVOOK4PAKZVCOGLRW6MYA, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.321859146 (551h34m8.192942191s) [noise=true , address=145.79.227.33:4871   , piece_id=XIGZHBN3OSRMCLNKJF2BDSRLKMGQ65P4725VGG5C2IE7CBIWGCTQ, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.322147725 (551h34m8.19323077s) [noise=true , address=145.79.227.6:4259    , piece_id=TMFLLJFNVKJTUCI5ZKJJNBCVQMUEWLDCQBICTELBIYIX3SGZ6HKQ, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.322600788 (551h34m8.193683823s) [noise=true , address=45.91.116.8:4047     , piece_id=5JMQ2UU2P2MHZ7BWVCNOYMKXVULSJBUUKSJD7JU64C6Q5Z4WU5ZA, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.323073278 (551h34m8.194156323s) [noise=true , address=145.79.227.2:4115    , piece_id=F6S5KEXEQU5ZVDALIBUZ5IU7XN4HFUEDT6IWBZ527K3C263CRIHQ, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.323253461 (551h34m8.194336506s) [noise=true , address=45.91.116.9:4263     , piece_id=UQSYALH4RNSWNNBFD7CFAUXIBQMW7OIX6YVK5XC6CXDYRUOJM5EQ, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.323441068 (551h34m8.194524113s) [noise=true , address=145.79.227.49:4338   , piece_id=INPHU2CKDWC6YCYRIX4M3CSERBGBNMAFUPDAP3MSRHVMLEBCQI6A, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.323739967 (551h34m8.194823012s) [noise=true , address=145.79.227.14:4837   , piece_id=EFGOFWFUS3Q75WXCTUKIWZORQF7S3AWQMMYVKNG2WQKSJN5N5KQA, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.323884763 (551h34m8.194967808s) [noise=true , address=145.79.227.48:4302   , piece_id=3IANNEIPANTNG4H6POCK7UP7GDZAG6FPUMLESOI7W5AGTTT3YOTQ, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.324003559 (551h34m8.195086594s) [noise=true , address=145.79.227.32:4834   , piece_id=EMPO6O5LRFUYOCHES23TBW53NI4DIPZ7J5DUHGJ6CQY2QPOEPAJA, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.324496458 (551h34m8.195579503s) [noise=true , address=145.79.227.42:4122   , piece_id=5CIB33YE4HYJWZKUHMVKMATGQJVWICKNGAIM6SCWFBH3LZFYIY3A, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.325091120 (551h34m8.196174166s) [noise=true , address=145.79.227.50:4375   , piece_id=5OGAHLED6Z7P3OLNAFWDX6LGQIAP6KGIKHSQHPAQXZ7WV3ARNZWQ, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.325640336 (551h34m8.196723382s) [noise=true , address=77.90.31.17:4770     , piece_id=YATVYQOR4LT6CROI4ODR27NHPXVRTYYOUO4NZ2CGVNCTUNMCSA2A, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:08.326083790 (551h34m8.197166835s) [noise=true , address=45.91.116.7:4010     , piece_id=BHWQROLFYGVWOGRGFACVJWVKAMFG46FU5XWCGA3TFNX7HFX2KRYA, seg_pos={0 0}, upload=483604]: Uploading piece...
Dec 28 15:03:09.948548634 (551h34m9.819631669s) [upload=483604]: Next returned nil segment
Dec 28 15:03:09.948595253 (551h34m9.819678298s) [upload=483604]: Waiting for error group managing segments...
Dec 28 15:03:10.333213440 (551h34m10.204296475s) [noise=true , address=145.79.227.16:4909   , piece_id=2PNWTZ2LFVNIIL4GH4QNV2BM6O6VBZUEVOOK4PAKZVCOGLRW6MYA, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.335359036 (551h34m10.206442081s) [noise=true , address=145.79.227.6:4259    , piece_id=TMFLLJFNVKJTUCI5ZKJJNBCVQMUEWLDCQBICTELBIYIX3SGZ6HKQ, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.335568084 (551h34m10.206651169s) [noise=true , address=145.79.227.42:4122   , piece_id=5CIB33YE4HYJWZKUHMVKMATGQJVWICKNGAIM6SCWFBH3LZFYIY3A, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.336124944 (551h34m10.207207979s) [noise=true , address=145.79.227.48:4302   , piece_id=3IANNEIPANTNG4H6POCK7UP7GDZAG6FPUMLESOI7W5AGTTT3YOTQ, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.336354120 (551h34m10.207437175s) [noise=true , address=145.79.227.14:4837   , piece_id=EFGOFWFUS3Q75WXCTUKIWZORQF7S3AWQMMYVKNG2WQKSJN5N5KQA, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.336925758 (551h34m10.208008844s) [noise=true , address=145.79.227.50:4375   , piece_id=5OGAHLED6Z7P3OLNAFWDX6LGQIAP6KGIKHSQHPAQXZ7WV3ARNZWQ, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.337959617 (551h34m10.209042662s) [noise=true , address=145.79.227.51:4411   , piece_id=ZMN6BDN5ILCAVQ7NPEMDTFIA3KU3XRDKDNMVGHDLNAREGPORUQ3Q, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.342216793 (551h34m10.213299829s) [noise=true , address=145.79.227.52:4445   , piece_id=AXI74ZBRV66TBGA6UYDSVGKK3JDW7OGK7N44QC4TSWL7RA6Y33UA, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.344627705 (551h34m10.21571075s) [noise=true , address=145.79.227.2:4115    , piece_id=F6S5KEXEQU5ZVDALIBUZ5IU7XN4HFUEDT6IWBZ527K3C263CRIHQ, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.351615851 (551h34m10.222698886s) [noise=true , address=145.79.227.32:4834   , piece_id=EMPO6O5LRFUYOCHES23TBW53NI4DIPZ7J5DUHGJ6CQY2QPOEPAJA, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.355210977 (551h34m10.226294022s) [noise=true , address=145.79.227.33:4871   , piece_id=XIGZHBN3OSRMCLNKJF2BDSRLKMGQ65P4725VGG5C2IE7CBIWGCTQ, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.614850434 (551h34m10.485933469s) [noise=true , address=77.90.31.17:4770     , piece_id=YATVYQOR4LT6CROI4ODR27NHPXVRTYYOUO4NZ2CGVNCTUNMCSA2A, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.661895035 (551h34m10.53297808s) [noise=true , address=45.91.116.7:4010     , piece_id=BHWQROLFYGVWOGRGFACVJWVKAMFG46FU5XWCGA3TFNX7HFX2KRYA, seg_pos={0 0}, upload=483604]: Done uploading piece. err: <nil>
Dec 28 15:03:10.661915413 (551h34m10.532998448s) [seg_pos={0 0}, upload=483604]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:03:10.662136604 (551h34m10.533219649s) [noise=true , address=45.91.116.8:4047     , piece_id=5JMQ2UU2P2MHZ7BWVCNOYMKXVULSJBUUKSJD7JU64C6Q5Z4WU5ZA, seg_pos={0 0}, upload=483604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 28 15:03:10.662143979 (551h34m10.533227014s) [noise=true , address=45.91.116.8:4047     , piece_id=5JMQ2UU2P2MHZ7BWVCNOYMKXVULSJBUUKSJD7JU64C6Q5Z4WU5ZA, seg_pos={0 0}, upload=483604]: Long tail cancellation due to optimal threshold
Dec 28 15:03:10.662394836 (551h34m10.533477901s) [noise=true , address=145.79.227.49:4338   , piece_id=INPHU2CKDWC6YCYRIX4M3CSERBGBNMAFUPDAP3MSRHVMLEBCQI6A, seg_pos={0 0}, upload=483604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 15:03:10.662476782 (551h34m10.533559827s) [noise=true , address=145.79.227.49:4338   , piece_id=INPHU2CKDWC6YCYRIX4M3CSERBGBNMAFUPDAP3MSRHVMLEBCQI6A, seg_pos={0 0}, upload=483604]: Long tail cancellation due to optimal threshold
Dec 28 15:03:10.662484176 (551h34m10.533567221s) [noise=true , address=45.91.116.9:4263     , piece_id=UQSYALH4RNSWNNBFD7CFAUXIBQMW7OIX6YVK5XC6CXDYRUOJM5EQ, seg_pos={0 0}, upload=483604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 28 15:03:10.662506248 (551h34m10.533589293s) [noise=true , address=45.91.116.9:4263     , piece_id=UQSYALH4RNSWNNBFD7CFAUXIBQMW7OIX6YVK5XC6CXDYRUOJM5EQ, seg_pos={0 0}, upload=483604]: Long tail cancellation due to optimal threshold
Dec 28 15:03:10.662559289 (551h34m10.533642335s) [seg_pos={0 0}, upload=483604]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:03:10.662653027 (551h34m10.533736072s) [upload=483604]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:03:10.662659981 (551h34m10.533743036s) [upload=483604]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:03:10.958915020 (551h34m10.829998055s) [upload=483604]: Done uploading segments...
Dec 28 15:03:10.958935739 (551h34m10.830018785s) [upload=483604]: Upload finished. err: <nil>
Duration: 30.002542934s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
45 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...X.X.XX.]
duration: 
4.675540519s
trace: 
https://jaeger.dev.storj.io/trace/4458ef4f93a32e3a
FAIL (2025-12-28 15:03:30 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.675589753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [............X...............X......X....X...................]
duration: 
25.332387079s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 15:03:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 25.332413139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [..........X...........X.....................................]
duration: 
16.819107684s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 15:03:11 UTC) 
orbiter warning: context deadline exceeded
Duration: 16.819136569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.X..XXXXX....X...X..X..X...X...X.X.........X.....X......X.X.]
duration: 
5.901823114s
trace: 
https://jaeger.dev.storj.io/trace/322b5eeca579c87e
FAIL (2025-12-28 15:02:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: JU5NPHWGWDFBJEGIXRSM3BY57M36SKG75X34IIBDVDSHN2A4PP2Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HRUEALYWQ7ZDC2CTK7VJSPCKJHOZA5SJOVZZOLA6ZPLU3Z3V4H7A) context deadline exceeded
======LOGS======
ec 28 15:01:37.131930242 (551h32m37.003013287s) [upload=483584]: Starting upload
Dec 28 15:01:37.132002599 (551h32m37.003085634s) [upload=483584]: Done starting upload
Dec 28 15:01:37.132073855 (551h32m37.0031569s) [upload=483584]: Uploading segments...
Dec 28 15:01:37.132189004 (551h32m37.003272039s) [upload=483584]: Got next segment. Inline: false
Dec 28 15:01:37.132231364 (551h32m37.003314399s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:01:37.132235452 (551h32m37.003318497s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:01:37.379181148 (551h32m37.250264193s) [seg_pos={0 0}, upload=483584]: Begin upload segment...
Dec 28 15:01:37.379245821 (551h32m37.250328866s) [seg_pos={0 0}, upload=483584]: Done begin upload segment.
Dec 28 15:01:37.379283533 (551h32m37.250366568s) [noise=true , address=77.90.31.14:4661     , piece_id=JSQIGEW3TJZXBZKDJSOL4J7KGLBTJLDBHKBOYIIUHSZUREEBCQNA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.379311185 (551h32m37.25039422s) [noise=true , address=145.79.227.50:4375   , piece_id=B6DPLL4JRPHFDCEWHGGFQOT7NGMNJJN4F3JQ4HMVVB2GR6BQ27ZA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.379378674 (551h32m37.250461719s) [noise=true , address=45.91.116.4:4940     , piece_id=JOQLWWZ6CNW6WNESLFH2W4Z3E2HZ734UPVWES5JY4C2WWHIHB7LQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.379851184 (551h32m37.250934229s) [noise=true , address=45.91.116.6:4973     , piece_id=3JHBQBZRTLL7R2E53Y32GNKTQF3AVSWDU2HMYVPSEFVW5GIJUCJQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.380302674 (551h32m37.251385919s) [noise=true , address=145.79.227.48:4301   , piece_id=AY7BPC3PBHE6AQGM3HIVNZNFT2LXKADCQ5HVB4ZP7DJG3Z2JMBAQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.380765986 (551h32m37.251849031s) [noise=true , address=145.79.227.51:4410   , piece_id=QSHGZSOHMK7D3DDCZGXABRGS5E4Z6DRBMDAATFWW4K7PR5FDHJ6Q, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.381101595 (551h32m37.25218464s) [noise=true , address=145.79.227.11:4729   , piece_id=5LLVBJYAQIMS7HQTYOSGGYO37HHE7NNKDHS5BPQYTOLBKQGDQ6XA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.381447925 (551h32m37.25253096s) [noise=true , address=45.91.116.13:4155    , piece_id=JU5NPHWGWDFBJEGIXRSM3BY57M36SKG75X34IIBDVDSHN2A4PP2Q, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.382214875 (551h32m37.25329791s) [noise=true , address=145.79.227.52:4445   , piece_id=Y2YE3SENV22RTKQBTAYYFJXJABHYRKPUKODGBGMSESPZAALLNLJA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.382762767 (551h32m37.253845803s) [noise=true , address=77.90.31.17:4773     , piece_id=QLRC7KMIF43V7TBLSC5WVCEJHFYON2LKTGIMWCLSTBNRFLOSSKTQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.384166331 (551h32m37.255249376s) [noise=true , address=145.79.227.36:4518   , piece_id=V5XKS6BKABHDNIRRR73AKAF4IPWTHUB2EBQ7CBWUTVCOE4RI6QWQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.385398497 (551h32m37.256481532s) [noise=true , address=145.79.227.14:4840   , piece_id=D6H7EHTGQJKS2GC2BYXSKPX6UXXD6TXHL4J46JRLWYSSKHIKJQIA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.386741444 (551h32m37.257824479s) [noise=true , address=77.90.31.11:4555     , piece_id=HRUEALYWQ7ZDC2CTK7VJSPCKJHOZA5SJOVZZOLA6ZPLU3Z3V4H7A, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.388198629 (551h32m37.259281664s) [noise=true , address=145.79.227.12:4765   , piece_id=5RF5F6HD5Y7SFHWRW72BRUMXXEEQRQVSYAYSHE77OIQARZWQQOUA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.389712381 (551h32m37.260795426s) [noise=true , address=145.79.227.27:4691   , piece_id=SBFPC2OGQ3DPVTYC73N5X2PGRZZHOLJSQI3N6OBPX6L2MLX2AW2Q, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.391389545 (551h32m37.26247259s) [noise=true , address=145.79.227.4:4186    , piece_id=4DFCA2VTGMSP6EO76IMNMNI2FXVGL4SNFNHJMKPKFL2XCIRBGPUQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:39.090791249 (551h32m38.961874285s) [upload=483584]: Next returned nil segment
Dec 28 15:01:39.090805466 (551h32m38.961888511s) [upload=483584]: Waiting for error group managing segments...
Dec 28 15:01:39.372264221 (551h32m39.243347266s) [noise=true , address=145.79.227.11:4729   , piece_id=5LLVBJYAQIMS7HQTYOSGGYO37HHE7NNKDHS5BPQYTOLBKQGDQ6XA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.382052429 (551h32m39.253135474s) [noise=true , address=145.79.227.50:4375   , piece_id=B6DPLL4JRPHFDCEWHGGFQOT7NGMNJJN4F3JQ4HMVVB2GR6BQ27ZA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.382735680 (551h32m39.253818726s) [noise=true , address=145.79.227.52:4445   , piece_id=Y2YE3SENV22RTKQBTAYYFJXJABHYRKPUKODGBGMSESPZAALLNLJA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.387203788 (551h32m39.258286833s) [noise=true , address=145.79.227.51:4410   , piece_id=QSHGZSOHMK7D3DDCZGXABRGS5E4Z6DRBMDAATFWW4K7PR5FDHJ6Q, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.391413314 (551h32m39.262496359s) [noise=true , address=145.79.227.4:4186    , piece_id=4DFCA2VTGMSP6EO76IMNMNI2FXVGL4SNFNHJMKPKFL2XCIRBGPUQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.393438019 (551h32m39.264521064s) [noise=true , address=145.79.227.48:4301   , piece_id=AY7BPC3PBHE6AQGM3HIVNZNFT2LXKADCQ5HVB4ZP7DJG3Z2JMBAQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.396869103 (551h32m39.267952138s) [noise=true , address=145.79.227.12:4765   , piece_id=5RF5F6HD5Y7SFHWRW72BRUMXXEEQRQVSYAYSHE77OIQARZWQQOUA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.397362242 (551h32m39.268445277s) [noise=true , address=145.79.227.36:4518   , piece_id=V5XKS6BKABHDNIRRR73AKAF4IPWTHUB2EBQ7CBWUTVCOE4RI6QWQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.398837430 (551h32m39.269920475s) [noise=true , address=145.79.227.27:4691   , piece_id=SBFPC2OGQ3DPVTYC73N5X2PGRZZHOLJSQI3N6OBPX6L2MLX2AW2Q, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.405609676 (551h32m39.276692721s) [noise=true , address=145.79.227.14:4840   , piece_id=D6H7EHTGQJKS2GC2BYXSKPX6UXXD6TXHL4J46JRLWYSSKHIKJQIA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.645420036 (551h32m39.516503071s) [noise=true , address=77.90.31.11:4555     , piece_id=HRUEALYWQ7ZDC2CTK7VJSPCKJHOZA5SJOVZZOLA6ZPLU3Z3V4H7A, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:40.352809142 (551h32m40.223892177s) [noise=true , address=45.91.116.13:4155    , piece_id=JU5NPHWGWDFBJEGIXRSM3BY57M36SKG75X34IIBDVDSHN2A4PP2Q, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:40.423121068 (551h32m40.294204113s) [noise=true , address=77.90.31.14:4661     , piece_id=JSQIGEW3TJZXBZKDJSOL4J7KGLBTJLDBHKBOYIIUHSZUREEBCQNA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:40.423146906 (551h32m40.294229941s) [seg_pos={0 0}, upload=483584]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:01:40.423458339 (551h32m40.294541395s) [noise=true , address=77.90.31.17:4773     , piece_id=QLRC7KMIF43V7TBLSC5WVCEJHFYON2LKTGIMWCLSTBNRFLOSSKTQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 28 15:01:40.423469501 (551h32m40.294552546s) [noise=true , address=77.90.31.17:4773     , piece_id=QLRC7KMIF43V7TBLSC5WVCEJHFYON2LKTGIMWCLSTBNRFLOSSKTQ, seg_pos={0 0}, upload=483584]: Long tail cancellation due to optimal threshold
Dec 28 15:01:40.423916161 (551h32m40.294999206s) [noise=true , address=45.91.116.6:4973     , piece_id=3JHBQBZRTLL7R2E53Y32GNKTQF3AVSWDU2HMYVPSEFVW5GIJUCJQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 15:01:40.423934476 (551h32m40.295017511s) [noise=true , address=45.91.116.6:4973     , piece_id=3JHBQBZRTLL7R2E53Y32GNKTQF3AVSWDU2HMYVPSEFVW5GIJUCJQ, seg_pos={0 0}, upload=483584]: Long tail cancellation due to optimal threshold
Dec 28 15:01:40.423942772 (551h32m40.295025817s) [noise=true , address=45.91.116.4:4940     , piece_id=JOQLWWZ6CNW6WNESLFH2W4Z3E2HZ734UPVWES5JY4C2WWHIHB7LQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:41264->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:41264->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->45.91.116.4:4940: use of closed network connection
Dec 28 15:01:40.423953993 (551h32m40.295037038s) [noise=true , address=45.91.116.4:4940     , piece_id=JOQLWWZ6CNW6WNESLFH2W4Z3E2HZ734UPVWES5JY4C2WWHIHB7LQ, seg_pos={0 0}, upload=483584]: Long tail cancellation due to optimal threshold
Dec 28 15:01:40.423974132 (551h32m40.295057167s) [seg_pos={0 0}, upload=483584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:01:40.424090343 (551h32m40.295173388s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:01:40.424096936 (551h32m40.295179981s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:01:40.716320397 (551h32m40.587403432s) [upload=483584]: Done uploading segments...
Dec 28 15:01:40.716359812 (551h32m40.587442857s) [upload=483584]: Upload finished. err: <nil>
Duration: 5.90184767s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [...........X...........X....................................]
duration: 
12.007003041s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 15:02:41 UTC) 
orbiter warning: context deadline exceeded
Duration: 12.007032566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [X............X...............X......X....X..................]
duration: 
42.450183695s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 15:02:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 42.450213121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...X.X.XX]
duration: 
5.002047882s
trace: 
https://jaeger.dev.storj.io/trace/4458ef4f93a32e3a
FAIL (2025-12-28 15:02:31 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002069613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [............X...........X...................................]
duration: 
17.571194081s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 15:02:27 UTC) 
orbiter warning: context deadline exceeded
Duration: 17.571218668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [X.X..XXXXX....X...X..X..X...X...X.X.........X.....X......X.X]
duration: 
30.003764814s
trace: 
https://jaeger.dev.storj.io/trace/322b5eeca579c87e
FAIL (2025-12-28 15:02:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: JU5NPHWGWDFBJEGIXRSM3BY57M36SKG75X34IIBDVDSHN2A4PP2Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HRUEALYWQ7ZDC2CTK7VJSPCKJHOZA5SJOVZZOLA6ZPLU3Z3V4H7A) context deadline exceeded
======LOGS======
ec 28 15:01:37.131930242 (551h32m37.003013287s) [upload=483584]: Starting upload
Dec 28 15:01:37.132002599 (551h32m37.003085634s) [upload=483584]: Done starting upload
Dec 28 15:01:37.132073855 (551h32m37.0031569s) [upload=483584]: Uploading segments...
Dec 28 15:01:37.132189004 (551h32m37.003272039s) [upload=483584]: Got next segment. Inline: false
Dec 28 15:01:37.132231364 (551h32m37.003314399s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:01:37.132235452 (551h32m37.003318497s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:01:37.379181148 (551h32m37.250264193s) [seg_pos={0 0}, upload=483584]: Begin upload segment...
Dec 28 15:01:37.379245821 (551h32m37.250328866s) [seg_pos={0 0}, upload=483584]: Done begin upload segment.
Dec 28 15:01:37.379283533 (551h32m37.250366568s) [noise=true , address=77.90.31.14:4661     , piece_id=JSQIGEW3TJZXBZKDJSOL4J7KGLBTJLDBHKBOYIIUHSZUREEBCQNA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.379311185 (551h32m37.25039422s) [noise=true , address=145.79.227.50:4375   , piece_id=B6DPLL4JRPHFDCEWHGGFQOT7NGMNJJN4F3JQ4HMVVB2GR6BQ27ZA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.379378674 (551h32m37.250461719s) [noise=true , address=45.91.116.4:4940     , piece_id=JOQLWWZ6CNW6WNESLFH2W4Z3E2HZ734UPVWES5JY4C2WWHIHB7LQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.379851184 (551h32m37.250934229s) [noise=true , address=45.91.116.6:4973     , piece_id=3JHBQBZRTLL7R2E53Y32GNKTQF3AVSWDU2HMYVPSEFVW5GIJUCJQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.380302674 (551h32m37.251385919s) [noise=true , address=145.79.227.48:4301   , piece_id=AY7BPC3PBHE6AQGM3HIVNZNFT2LXKADCQ5HVB4ZP7DJG3Z2JMBAQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.380765986 (551h32m37.251849031s) [noise=true , address=145.79.227.51:4410   , piece_id=QSHGZSOHMK7D3DDCZGXABRGS5E4Z6DRBMDAATFWW4K7PR5FDHJ6Q, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.381101595 (551h32m37.25218464s) [noise=true , address=145.79.227.11:4729   , piece_id=5LLVBJYAQIMS7HQTYOSGGYO37HHE7NNKDHS5BPQYTOLBKQGDQ6XA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.381447925 (551h32m37.25253096s) [noise=true , address=45.91.116.13:4155    , piece_id=JU5NPHWGWDFBJEGIXRSM3BY57M36SKG75X34IIBDVDSHN2A4PP2Q, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.382214875 (551h32m37.25329791s) [noise=true , address=145.79.227.52:4445   , piece_id=Y2YE3SENV22RTKQBTAYYFJXJABHYRKPUKODGBGMSESPZAALLNLJA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.382762767 (551h32m37.253845803s) [noise=true , address=77.90.31.17:4773     , piece_id=QLRC7KMIF43V7TBLSC5WVCEJHFYON2LKTGIMWCLSTBNRFLOSSKTQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.384166331 (551h32m37.255249376s) [noise=true , address=145.79.227.36:4518   , piece_id=V5XKS6BKABHDNIRRR73AKAF4IPWTHUB2EBQ7CBWUTVCOE4RI6QWQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.385398497 (551h32m37.256481532s) [noise=true , address=145.79.227.14:4840   , piece_id=D6H7EHTGQJKS2GC2BYXSKPX6UXXD6TXHL4J46JRLWYSSKHIKJQIA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.386741444 (551h32m37.257824479s) [noise=true , address=77.90.31.11:4555     , piece_id=HRUEALYWQ7ZDC2CTK7VJSPCKJHOZA5SJOVZZOLA6ZPLU3Z3V4H7A, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.388198629 (551h32m37.259281664s) [noise=true , address=145.79.227.12:4765   , piece_id=5RF5F6HD5Y7SFHWRW72BRUMXXEEQRQVSYAYSHE77OIQARZWQQOUA, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.389712381 (551h32m37.260795426s) [noise=true , address=145.79.227.27:4691   , piece_id=SBFPC2OGQ3DPVTYC73N5X2PGRZZHOLJSQI3N6OBPX6L2MLX2AW2Q, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:37.391389545 (551h32m37.26247259s) [noise=true , address=145.79.227.4:4186    , piece_id=4DFCA2VTGMSP6EO76IMNMNI2FXVGL4SNFNHJMKPKFL2XCIRBGPUQ, seg_pos={0 0}, upload=483584]: Uploading piece...
Dec 28 15:01:39.090791249 (551h32m38.961874285s) [upload=483584]: Next returned nil segment
Dec 28 15:01:39.090805466 (551h32m38.961888511s) [upload=483584]: Waiting for error group managing segments...
Dec 28 15:01:39.372264221 (551h32m39.243347266s) [noise=true , address=145.79.227.11:4729   , piece_id=5LLVBJYAQIMS7HQTYOSGGYO37HHE7NNKDHS5BPQYTOLBKQGDQ6XA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.382052429 (551h32m39.253135474s) [noise=true , address=145.79.227.50:4375   , piece_id=B6DPLL4JRPHFDCEWHGGFQOT7NGMNJJN4F3JQ4HMVVB2GR6BQ27ZA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.382735680 (551h32m39.253818726s) [noise=true , address=145.79.227.52:4445   , piece_id=Y2YE3SENV22RTKQBTAYYFJXJABHYRKPUKODGBGMSESPZAALLNLJA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.387203788 (551h32m39.258286833s) [noise=true , address=145.79.227.51:4410   , piece_id=QSHGZSOHMK7D3DDCZGXABRGS5E4Z6DRBMDAATFWW4K7PR5FDHJ6Q, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.391413314 (551h32m39.262496359s) [noise=true , address=145.79.227.4:4186    , piece_id=4DFCA2VTGMSP6EO76IMNMNI2FXVGL4SNFNHJMKPKFL2XCIRBGPUQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.393438019 (551h32m39.264521064s) [noise=true , address=145.79.227.48:4301   , piece_id=AY7BPC3PBHE6AQGM3HIVNZNFT2LXKADCQ5HVB4ZP7DJG3Z2JMBAQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.396869103 (551h32m39.267952138s) [noise=true , address=145.79.227.12:4765   , piece_id=5RF5F6HD5Y7SFHWRW72BRUMXXEEQRQVSYAYSHE77OIQARZWQQOUA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.397362242 (551h32m39.268445277s) [noise=true , address=145.79.227.36:4518   , piece_id=V5XKS6BKABHDNIRRR73AKAF4IPWTHUB2EBQ7CBWUTVCOE4RI6QWQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.398837430 (551h32m39.269920475s) [noise=true , address=145.79.227.27:4691   , piece_id=SBFPC2OGQ3DPVTYC73N5X2PGRZZHOLJSQI3N6OBPX6L2MLX2AW2Q, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.405609676 (551h32m39.276692721s) [noise=true , address=145.79.227.14:4840   , piece_id=D6H7EHTGQJKS2GC2BYXSKPX6UXXD6TXHL4J46JRLWYSSKHIKJQIA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:39.645420036 (551h32m39.516503071s) [noise=true , address=77.90.31.11:4555     , piece_id=HRUEALYWQ7ZDC2CTK7VJSPCKJHOZA5SJOVZZOLA6ZPLU3Z3V4H7A, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:40.352809142 (551h32m40.223892177s) [noise=true , address=45.91.116.13:4155    , piece_id=JU5NPHWGWDFBJEGIXRSM3BY57M36SKG75X34IIBDVDSHN2A4PP2Q, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:40.423121068 (551h32m40.294204113s) [noise=true , address=77.90.31.14:4661     , piece_id=JSQIGEW3TJZXBZKDJSOL4J7KGLBTJLDBHKBOYIIUHSZUREEBCQNA, seg_pos={0 0}, upload=483584]: Done uploading piece. err: <nil>
Dec 28 15:01:40.423146906 (551h32m40.294229941s) [seg_pos={0 0}, upload=483584]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:01:40.423458339 (551h32m40.294541395s) [noise=true , address=77.90.31.17:4773     , piece_id=QLRC7KMIF43V7TBLSC5WVCEJHFYON2LKTGIMWCLSTBNRFLOSSKTQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 28 15:01:40.423469501 (551h32m40.294552546s) [noise=true , address=77.90.31.17:4773     , piece_id=QLRC7KMIF43V7TBLSC5WVCEJHFYON2LKTGIMWCLSTBNRFLOSSKTQ, seg_pos={0 0}, upload=483584]: Long tail cancellation due to optimal threshold
Dec 28 15:01:40.423916161 (551h32m40.294999206s) [noise=true , address=45.91.116.6:4973     , piece_id=3JHBQBZRTLL7R2E53Y32GNKTQF3AVSWDU2HMYVPSEFVW5GIJUCJQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 15:01:40.423934476 (551h32m40.295017511s) [noise=true , address=45.91.116.6:4973     , piece_id=3JHBQBZRTLL7R2E53Y32GNKTQF3AVSWDU2HMYVPSEFVW5GIJUCJQ, seg_pos={0 0}, upload=483584]: Long tail cancellation due to optimal threshold
Dec 28 15:01:40.423942772 (551h32m40.295025817s) [noise=true , address=45.91.116.4:4940     , piece_id=JOQLWWZ6CNW6WNESLFH2W4Z3E2HZ734UPVWES5JY4C2WWHIHB7LQ, seg_pos={0 0}, upload=483584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:41264->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:41264->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->45.91.116.4:4940: use of closed network connection
Dec 28 15:01:40.423953993 (551h32m40.295037038s) [noise=true , address=45.91.116.4:4940     , piece_id=JOQLWWZ6CNW6WNESLFH2W4Z3E2HZ734UPVWES5JY4C2WWHIHB7LQ, seg_pos={0 0}, upload=483584]: Long tail cancellation due to optimal threshold
Dec 28 15:01:40.423974132 (551h32m40.295057167s) [seg_pos={0 0}, upload=483584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:01:40.424090343 (551h32m40.295173388s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:01:40.424096936 (551h32m40.295179981s) [upload=483584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:01:40.716320397 (551h32m40.587403432s) [upload=483584]: Done uploading segments...
Dec 28 15:01:40.716359812 (551h32m40.587442857s) [upload=483584]: Upload finished. err: <nil>
Duration: 30.003791936s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X...........X..................................]
duration: 
13.300029155s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 15:02:04 UTC) 
orbiter warning: context deadline exceeded
Duration: 13.300211722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.X............X...............X......X....X.................]
duration: 
26.095965621s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 15:01:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 26.096022899s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [..............X...........X.................................]
duration: 
10.572214328s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 15:01:35 UTC) 
orbiter warning: context deadline exceeded
Duration: 10.572239266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [..X............X...............X......X....X................]
duration: 
47.449639014s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 15:01:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 47.449706502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X......X.]
duration: 
5.872729395s
trace: 
https://jaeger.dev.storj.io/trace/30e2cb0f4fb4812c
FAIL (2025-12-28 15:01:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 15:00:47.558055944 (551h31m47.429138979s) [upload=483570]: Starting upload
Dec 28 15:00:47.558135496 (551h31m47.429218531s) [upload=483570]: Done starting upload
Dec 28 15:00:47.558192675 (551h31m47.42927571s) [upload=483570]: Uploading segments...
Dec 28 15:00:47.558359533 (551h31m47.429442578s) [upload=483570]: Got next segment. Inline: false
Dec 28 15:00:47.558459904 (551h31m47.429542939s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:00:47.558467689 (551h31m47.429550734s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:00:47.822984652 (551h31m47.694067687s) [seg_pos={0 0}, upload=483570]: Begin upload segment...
Dec 28 15:00:47.823039285 (551h31m47.69412232s) [seg_pos={0 0}, upload=483570]: Done begin upload segment.
Dec 28 15:00:47.823124719 (551h31m47.694207754s) [noise=true , address=77.90.31.10:4520     , piece_id=5YRDY3IFPX3YIKPGHZ5QYJQW5OQGKMCPGA7UYDMXQMGMEDSHZN7Q, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.823133565 (551h31m47.69421662s) [noise=true , address=77.90.31.19:4842     , piece_id=42HDHG3XK4M7LGDJJWQB2CW2WXULDBLM5BWUTH26YHSQNQD75NWQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.823754418 (551h31m47.694837463s) [noise=true , address=77.90.31.11:4554     , piece_id=ETUPGOUGQDIMVCOHSKUXJESJUK4KT4LLOLJGHDJH4A2VMX5777VA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.824159158 (551h31m47.695242203s) [noise=true , address=145.79.227.18:4007   , piece_id=7Y3LSSRON3S4BIKN3TK63KIIKELPHW3RNDHKEPCXU4ODQVD7VMTQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.824492262 (551h31m47.695575297s) [noise=true , address=145.79.227.11:4730   , piece_id=ZNBXUMXSVTDU2PCQ56ONCW5IJSO65LTAZ6P6UVVEGCEPOLEP3OSA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.825986979 (551h31m47.697070014s) [noise=true , address=145.79.227.37:4567   , piece_id=FUVW5GMKOSIXBTCHIKXIINSWPB7V667T4YXGN2OW5HAY4G64JQNQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.826389665 (551h31m47.69747271s) [noise=true , address=77.90.31.21:4877     , piece_id=Z5TNET5SPQDMQLAYLXYL6TD3YMGSRIWC3D7AZSJDEWNJUW24YOUA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.827697566 (551h31m47.698780601s) [noise=true , address=45.91.116.14:4189    , piece_id=GQ22TZFKYTIOT6DAHEQKNCNNN32HRZKMVBPEMPJTXBSLPS335CPQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.828046089 (551h31m47.699129124s) [noise=true , address=45.91.116.8:4045     , piece_id=FXGVCROTYU5Y2CVLQ224LQ2S6YVUGK37I3U5RVWSM45HM5QZTHEA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.828336362 (551h31m47.699419407s) [noise=true , address=145.79.227.15:4875   , piece_id=5QZM3YB4NDQUKOUHFHI3BT5IF4FUDH7PFXCRWFKSSX7RB5VPJYXQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.830455778 (551h31m47.701538813s) [noise=true , address=77.90.31.13:4629     , piece_id=YNCWQJ4TVLGBXQO3DJMMQGYJRKU4TIUJCK6E2UXNY63QIJZDWMLQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.840098739 (551h31m47.711181784s) [noise=true , address=45.91.116.12:4228    , piece_id=LFS2I6NXSCMJWCRO6BJNA45Y77I52XGGF4BJFN6R2ADUJOCAGL2A, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.841339603 (551h31m47.712422648s) [noise=true , address=45.91.116.11:4118    , piece_id=4NJNXACF5I3YMH4FGAFBLUEWZSGZVS7DDAKAMRPQHNEBAC6ASGMA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.842743506 (551h31m47.713826551s) [noise=true , address=145.79.227.40:4050   , piece_id=3HZYPV2KTCUXQ6MWZ6KIF37ERMHKJQJYS2HGR2QVF5HF24XHXUFQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.843506018 (551h31m47.714589063s) [noise=true , address=145.79.227.14:4840   , piece_id=EPCQ4KVWU4U4HOVMUCZDARCWK6MU3SWEI44KRXAUPE4PPQUT446A, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.844014156 (551h31m47.715097191s) [noise=true , address=77.90.31.15:4697     , piece_id=2AN6AIAN3DCI3BQDPWCDRHDCWU5I4EK4PQYHRKDIOJXAU6TR5KHQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:49.551230892 (551h31m49.422313937s) [upload=483570]: Next returned nil segment
Dec 28 15:00:49.551246572 (551h31m49.422329617s) [upload=483570]: Waiting for error group managing segments...
Dec 28 15:00:49.835363838 (551h31m49.706446883s) [noise=true , address=145.79.227.15:4875   , piece_id=5QZM3YB4NDQUKOUHFHI3BT5IF4FUDH7PFXCRWFKSSX7RB5VPJYXQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.835786804 (551h31m49.706869849s) [noise=true , address=145.79.227.18:4007   , piece_id=7Y3LSSRON3S4BIKN3TK63KIIKELPHW3RNDHKEPCXU4ODQVD7VMTQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.846210412 (551h31m49.717293457s) [noise=true , address=145.79.227.14:4840   , piece_id=EPCQ4KVWU4U4HOVMUCZDARCWK6MU3SWEI44KRXAUPE4PPQUT446A, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.847609727 (551h31m49.718692772s) [noise=true , address=145.79.227.11:4730   , piece_id=ZNBXUMXSVTDU2PCQ56ONCW5IJSO65LTAZ6P6UVVEGCEPOLEP3OSA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.857502274 (551h31m49.728585309s) [noise=true , address=145.79.227.37:4567   , piece_id=FUVW5GMKOSIXBTCHIKXIINSWPB7V667T4YXGN2OW5HAY4G64JQNQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.093767362 (551h31m49.964850397s) [noise=true , address=77.90.31.11:4554     , piece_id=ETUPGOUGQDIMVCOHSKUXJESJUK4KT4LLOLJGHDJH4A2VMX5777VA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.096612049 (551h31m49.967695094s) [noise=true , address=77.90.31.10:4520     , piece_id=5YRDY3IFPX3YIKPGHZ5QYJQW5OQGKMCPGA7UYDMXQMGMEDSHZN7Q, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.110065095 (551h31m49.981148131s) [noise=true , address=77.90.31.21:4877     , piece_id=Z5TNET5SPQDMQLAYLXYL6TD3YMGSRIWC3D7AZSJDEWNJUW24YOUA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.177942295 (551h31m50.049025341s) [noise=true , address=45.91.116.14:4189    , piece_id=GQ22TZFKYTIOT6DAHEQKNCNNN32HRZKMVBPEMPJTXBSLPS335CPQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.230217095 (551h31m50.10130014s) [noise=true , address=45.91.116.12:4228    , piece_id=LFS2I6NXSCMJWCRO6BJNA45Y77I52XGGF4BJFN6R2ADUJOCAGL2A, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.718697303 (551h31m50.589780338s) [noise=true , address=77.90.31.15:4697     , piece_id=2AN6AIAN3DCI3BQDPWCDRHDCWU5I4EK4PQYHRKDIOJXAU6TR5KHQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.769105760 (551h31m50.640188805s) [noise=true , address=77.90.31.19:4842     , piece_id=42HDHG3XK4M7LGDJJWQB2CW2WXULDBLM5BWUTH26YHSQNQD75NWQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:51.219125492 (551h31m51.090208537s) [noise=true , address=45.91.116.11:4118    , piece_id=4NJNXACF5I3YMH4FGAFBLUEWZSGZVS7DDAKAMRPQHNEBAC6ASGMA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:51.219212407 (551h31m51.090295452s) [seg_pos={0 0}, upload=483570]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:00:51.219429851 (551h31m51.090512896s) [noise=true , address=45.91.116.8:4045     , piece_id=FXGVCROTYU5Y2CVLQ224LQ2S6YVUGK37I3U5RVWSM45HM5QZTHEA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 28 15:00:51.219438719 (551h31m51.090521764s) [noise=true , address=45.91.116.8:4045     , piece_id=FXGVCROTYU5Y2CVLQ224LQ2S6YVUGK37I3U5RVWSM45HM5QZTHEA, seg_pos={0 0}, upload=483570]: Long tail cancellation due to optimal threshold
Dec 28 15:00:51.219675118 (551h31m51.090758163s) [noise=true , address=77.90.31.13:4629     , piece_id=YNCWQJ4TVLGBXQO3DJMMQGYJRKU4TIUJCK6E2UXNY63QIJZDWMLQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF
Dec 28 15:00:51.219684036 (551h31m51.090767081s) [noise=true , address=77.90.31.13:4629     , piece_id=YNCWQJ4TVLGBXQO3DJMMQGYJRKU4TIUJCK6E2UXNY63QIJZDWMLQ, seg_pos={0 0}, upload=483570]: Long tail cancellation due to optimal threshold
Dec 28 15:00:51.219791931 (551h31m51.090875006s) [noise=true , address=145.79.227.40:4050   , piece_id=3HZYPV2KTCUXQ6MWZ6KIF37ERMHKJQJYS2HGR2QVF5HF24XHXUFQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 28 15:00:51.219830905 (551h31m51.09091397s) [noise=true , address=145.79.227.40:4050   , piece_id=3HZYPV2KTCUXQ6MWZ6KIF37ERMHKJQJYS2HGR2QVF5HF24XHXUFQ, seg_pos={0 0}, upload=483570]: Long tail cancellation due to optimal threshold
Dec 28 15:00:51.219859159 (551h31m51.090942204s) [seg_pos={0 0}, upload=483570]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:00:51.220004746 (551h31m51.091087791s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:00:51.220012120 (551h31m51.091095165s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:00:51.511776418 (551h31m51.382859493s) [upload=483570]: Done uploading segments...
Dec 28 15:00:51.511803850 (551h31m51.382886895s) [upload=483570]: Upload finished. err: <nil>
Duration: 5.872745115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X...........X................................]
duration: 
13.584582463s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 15:01:21 UTC) 
orbiter warning: context deadline exceeded
Duration: 13.584747375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...X.X.X]
duration: 
5.001146248s
trace: 
https://jaeger.dev.storj.io/trace/2719207c520478d8
FAIL (2025-12-28 15:01:19 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001171666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [X.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X......X]
duration: 
30.004005101s
trace: 
https://jaeger.dev.storj.io/trace/30e2cb0f4fb4812c
FAIL (2025-12-28 15:01:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 15:00:47.558055944 (551h31m47.429138979s) [upload=483570]: Starting upload
Dec 28 15:00:47.558135496 (551h31m47.429218531s) [upload=483570]: Done starting upload
Dec 28 15:00:47.558192675 (551h31m47.42927571s) [upload=483570]: Uploading segments...
Dec 28 15:00:47.558359533 (551h31m47.429442578s) [upload=483570]: Got next segment. Inline: false
Dec 28 15:00:47.558459904 (551h31m47.429542939s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:00:47.558467689 (551h31m47.429550734s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:00:47.822984652 (551h31m47.694067687s) [seg_pos={0 0}, upload=483570]: Begin upload segment...
Dec 28 15:00:47.823039285 (551h31m47.69412232s) [seg_pos={0 0}, upload=483570]: Done begin upload segment.
Dec 28 15:00:47.823124719 (551h31m47.694207754s) [noise=true , address=77.90.31.10:4520     , piece_id=5YRDY3IFPX3YIKPGHZ5QYJQW5OQGKMCPGA7UYDMXQMGMEDSHZN7Q, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.823133565 (551h31m47.69421662s) [noise=true , address=77.90.31.19:4842     , piece_id=42HDHG3XK4M7LGDJJWQB2CW2WXULDBLM5BWUTH26YHSQNQD75NWQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.823754418 (551h31m47.694837463s) [noise=true , address=77.90.31.11:4554     , piece_id=ETUPGOUGQDIMVCOHSKUXJESJUK4KT4LLOLJGHDJH4A2VMX5777VA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.824159158 (551h31m47.695242203s) [noise=true , address=145.79.227.18:4007   , piece_id=7Y3LSSRON3S4BIKN3TK63KIIKELPHW3RNDHKEPCXU4ODQVD7VMTQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.824492262 (551h31m47.695575297s) [noise=true , address=145.79.227.11:4730   , piece_id=ZNBXUMXSVTDU2PCQ56ONCW5IJSO65LTAZ6P6UVVEGCEPOLEP3OSA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.825986979 (551h31m47.697070014s) [noise=true , address=145.79.227.37:4567   , piece_id=FUVW5GMKOSIXBTCHIKXIINSWPB7V667T4YXGN2OW5HAY4G64JQNQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.826389665 (551h31m47.69747271s) [noise=true , address=77.90.31.21:4877     , piece_id=Z5TNET5SPQDMQLAYLXYL6TD3YMGSRIWC3D7AZSJDEWNJUW24YOUA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.827697566 (551h31m47.698780601s) [noise=true , address=45.91.116.14:4189    , piece_id=GQ22TZFKYTIOT6DAHEQKNCNNN32HRZKMVBPEMPJTXBSLPS335CPQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.828046089 (551h31m47.699129124s) [noise=true , address=45.91.116.8:4045     , piece_id=FXGVCROTYU5Y2CVLQ224LQ2S6YVUGK37I3U5RVWSM45HM5QZTHEA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.828336362 (551h31m47.699419407s) [noise=true , address=145.79.227.15:4875   , piece_id=5QZM3YB4NDQUKOUHFHI3BT5IF4FUDH7PFXCRWFKSSX7RB5VPJYXQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.830455778 (551h31m47.701538813s) [noise=true , address=77.90.31.13:4629     , piece_id=YNCWQJ4TVLGBXQO3DJMMQGYJRKU4TIUJCK6E2UXNY63QIJZDWMLQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.840098739 (551h31m47.711181784s) [noise=true , address=45.91.116.12:4228    , piece_id=LFS2I6NXSCMJWCRO6BJNA45Y77I52XGGF4BJFN6R2ADUJOCAGL2A, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.841339603 (551h31m47.712422648s) [noise=true , address=45.91.116.11:4118    , piece_id=4NJNXACF5I3YMH4FGAFBLUEWZSGZVS7DDAKAMRPQHNEBAC6ASGMA, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.842743506 (551h31m47.713826551s) [noise=true , address=145.79.227.40:4050   , piece_id=3HZYPV2KTCUXQ6MWZ6KIF37ERMHKJQJYS2HGR2QVF5HF24XHXUFQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.843506018 (551h31m47.714589063s) [noise=true , address=145.79.227.14:4840   , piece_id=EPCQ4KVWU4U4HOVMUCZDARCWK6MU3SWEI44KRXAUPE4PPQUT446A, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:47.844014156 (551h31m47.715097191s) [noise=true , address=77.90.31.15:4697     , piece_id=2AN6AIAN3DCI3BQDPWCDRHDCWU5I4EK4PQYHRKDIOJXAU6TR5KHQ, seg_pos={0 0}, upload=483570]: Uploading piece...
Dec 28 15:00:49.551230892 (551h31m49.422313937s) [upload=483570]: Next returned nil segment
Dec 28 15:00:49.551246572 (551h31m49.422329617s) [upload=483570]: Waiting for error group managing segments...
Dec 28 15:00:49.835363838 (551h31m49.706446883s) [noise=true , address=145.79.227.15:4875   , piece_id=5QZM3YB4NDQUKOUHFHI3BT5IF4FUDH7PFXCRWFKSSX7RB5VPJYXQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.835786804 (551h31m49.706869849s) [noise=true , address=145.79.227.18:4007   , piece_id=7Y3LSSRON3S4BIKN3TK63KIIKELPHW3RNDHKEPCXU4ODQVD7VMTQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.846210412 (551h31m49.717293457s) [noise=true , address=145.79.227.14:4840   , piece_id=EPCQ4KVWU4U4HOVMUCZDARCWK6MU3SWEI44KRXAUPE4PPQUT446A, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.847609727 (551h31m49.718692772s) [noise=true , address=145.79.227.11:4730   , piece_id=ZNBXUMXSVTDU2PCQ56ONCW5IJSO65LTAZ6P6UVVEGCEPOLEP3OSA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:49.857502274 (551h31m49.728585309s) [noise=true , address=145.79.227.37:4567   , piece_id=FUVW5GMKOSIXBTCHIKXIINSWPB7V667T4YXGN2OW5HAY4G64JQNQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.093767362 (551h31m49.964850397s) [noise=true , address=77.90.31.11:4554     , piece_id=ETUPGOUGQDIMVCOHSKUXJESJUK4KT4LLOLJGHDJH4A2VMX5777VA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.096612049 (551h31m49.967695094s) [noise=true , address=77.90.31.10:4520     , piece_id=5YRDY3IFPX3YIKPGHZ5QYJQW5OQGKMCPGA7UYDMXQMGMEDSHZN7Q, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.110065095 (551h31m49.981148131s) [noise=true , address=77.90.31.21:4877     , piece_id=Z5TNET5SPQDMQLAYLXYL6TD3YMGSRIWC3D7AZSJDEWNJUW24YOUA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.177942295 (551h31m50.049025341s) [noise=true , address=45.91.116.14:4189    , piece_id=GQ22TZFKYTIOT6DAHEQKNCNNN32HRZKMVBPEMPJTXBSLPS335CPQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.230217095 (551h31m50.10130014s) [noise=true , address=45.91.116.12:4228    , piece_id=LFS2I6NXSCMJWCRO6BJNA45Y77I52XGGF4BJFN6R2ADUJOCAGL2A, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.718697303 (551h31m50.589780338s) [noise=true , address=77.90.31.15:4697     , piece_id=2AN6AIAN3DCI3BQDPWCDRHDCWU5I4EK4PQYHRKDIOJXAU6TR5KHQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:50.769105760 (551h31m50.640188805s) [noise=true , address=77.90.31.19:4842     , piece_id=42HDHG3XK4M7LGDJJWQB2CW2WXULDBLM5BWUTH26YHSQNQD75NWQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:51.219125492 (551h31m51.090208537s) [noise=true , address=45.91.116.11:4118    , piece_id=4NJNXACF5I3YMH4FGAFBLUEWZSGZVS7DDAKAMRPQHNEBAC6ASGMA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: <nil>
Dec 28 15:00:51.219212407 (551h31m51.090295452s) [seg_pos={0 0}, upload=483570]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:00:51.219429851 (551h31m51.090512896s) [noise=true , address=45.91.116.8:4045     , piece_id=FXGVCROTYU5Y2CVLQ224LQ2S6YVUGK37I3U5RVWSM45HM5QZTHEA, seg_pos={0 0}, upload=483570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 28 15:00:51.219438719 (551h31m51.090521764s) [noise=true , address=45.91.116.8:4045     , piece_id=FXGVCROTYU5Y2CVLQ224LQ2S6YVUGK37I3U5RVWSM45HM5QZTHEA, seg_pos={0 0}, upload=483570]: Long tail cancellation due to optimal threshold
Dec 28 15:00:51.219675118 (551h31m51.090758163s) [noise=true , address=77.90.31.13:4629     , piece_id=YNCWQJ4TVLGBXQO3DJMMQGYJRKU4TIUJCK6E2UXNY63QIJZDWMLQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF
Dec 28 15:00:51.219684036 (551h31m51.090767081s) [noise=true , address=77.90.31.13:4629     , piece_id=YNCWQJ4TVLGBXQO3DJMMQGYJRKU4TIUJCK6E2UXNY63QIJZDWMLQ, seg_pos={0 0}, upload=483570]: Long tail cancellation due to optimal threshold
Dec 28 15:00:51.219791931 (551h31m51.090875006s) [noise=true , address=145.79.227.40:4050   , piece_id=3HZYPV2KTCUXQ6MWZ6KIF37ERMHKJQJYS2HGR2QVF5HF24XHXUFQ, seg_pos={0 0}, upload=483570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 28 15:00:51.219830905 (551h31m51.09091397s) [noise=true , address=145.79.227.40:4050   , piece_id=3HZYPV2KTCUXQ6MWZ6KIF37ERMHKJQJYS2HGR2QVF5HF24XHXUFQ, seg_pos={0 0}, upload=483570]: Long tail cancellation due to optimal threshold
Dec 28 15:00:51.219859159 (551h31m51.090942204s) [seg_pos={0 0}, upload=483570]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:00:51.220004746 (551h31m51.091087791s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:00:51.220012120 (551h31m51.091095165s) [upload=483570]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:00:51.511776418 (551h31m51.382859493s) [upload=483570]: Done uploading segments...
Dec 28 15:00:51.511803850 (551h31m51.382886895s) [upload=483570]: Upload finished. err: <nil>
Duration: 30.004028716s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [................X...........X...............................]
duration: 
14.141162299s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 15:00:37 UTC) 
orbiter warning: context deadline exceeded
Duration: 14.141238173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...X.X.]
duration: 
4.017342289s
trace: 
https://jaeger.dev.storj.io/trace/694db1d4bf5637c0
FAIL (2025-12-28 15:00:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.017372946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [...X............X...............X......X....X...............]
duration: 
43.493778041s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 15:00:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 43.493795114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X......]
duration: 
5.752721541s
trace: 
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:59:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
 28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 5.752743732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [X...X............X...............X......X....X..............]
duration: 
45.818825234s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:59:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 45.818836536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
47 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...X.X]
duration: 
5.000555962s
trace: 
https://jaeger.dev.storj.io/trace/694db1d4bf5637c0
FAIL (2025-12-28 14:59:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000578224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [.................X...........X..............................]
duration: 
12.35321945s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:59:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 12.353245049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [X...........................................................]
duration: 
10.140356318s
trace: 
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:59:24 UTC) 
orbiter warning: context deadline exceeded
Duration: 10.140382017s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.X..........................................................]
duration: 
21.375357976s
trace: 
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:59:05 UTC) 
orbiter warning: context deadline exceeded
Duration: 21.375416557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.X...X............X...............X......X....X.............]
duration: 
53.548953145s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:58:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 53.548969127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
47 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...X.]
duration: 
4.229894012s
trace: 
https://jaeger.dev.storj.io/trace/4ca7bbd6b83a9ef9
FAIL (2025-12-28 14:58:44 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.229921425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [..................X...........X.............................]
duration: 
15.140651023s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:58:44 UTC) 
orbiter warning: context deadline exceeded
Duration: 15.140678274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X.....]
duration: 
5.943231293s
trace: 
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:58:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
 28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 5.9432556s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..X.........................................................]
duration: 
16.08430973s
trace: 
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:58:10 UTC) 
orbiter warning: context deadline exceeded
Duration: 16.084339838s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [..X...X............X...............X......X....X............]
duration: 
52.688407782s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:57:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 52.688435065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...X]
duration: 
5.001457807s
trace: 
https://jaeger.dev.storj.io/trace/4ca7bbd6b83a9ef9
FAIL (2025-12-28 14:57:54 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001490228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X....]
duration: 
5.922091391s
trace: 
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:57:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
 28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 5.922120236s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [...................X...........X............................]
duration: 
15.807646155s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:57:46 UTC) 
orbiter warning: context deadline exceeded
Duration: 15.807805288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [....................X...........X...........................]
duration: 
15.054127699s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:57:25 UTC) 
orbiter warning: context deadline exceeded
Duration: 15.054150692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...X........................................................]
duration: 
19.931503425s
trace: 
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:57:19 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.931541768s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
47 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...]
duration: 
5.000661272s
trace: 
https://jaeger.dev.storj.io/trace/131df032dcbcf2b
FAIL (2025-12-28 14:57:18 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000688684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [...X...X............X...............X......X....X...........]
duration: 
54.623188679s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:56:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 54.623216954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration: 
9.330310076s
trace: 
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:56:47 UTC) 
orbiter warning: context deadline exceeded
Duration: 9.330336716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X...]
duration: 
5.825546642s
trace: 
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:56:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
 28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 5.825572602s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
3 out of 60 is failed (threshold=3) [X....................X...........X..........................]
duration: 
14.94163699s
trace: 
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:56:35 UTC) 
orbiter warning: context deadline exceeded
Duration: 14.941753693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration: 
10.601502772s
trace: 
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:56:32 UTC) 
orbiter warning: context deadline exceeded
Duration: 10.601529883s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
48 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX..]
duration: 
5.000640303s
trace: 
https://jaeger.dev.storj.io/trace/131df032dcbcf2b
FAIL (2025-12-28 14:56:20 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000658418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X..]
duration: 
6.085860922s
trace: 
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:55:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
 28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 6.085891018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [....X...X............X...............X......X....X..........]
duration: 
55.514633917s
trace: 
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:55:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515    , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966     , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608   , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931   , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947   , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010   , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867     , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585     , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816     , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801     , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650     , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794   , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212   , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075   , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889     , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711    , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688     , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 55.514664916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
48 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX.]
duration: 
3.544103916s
trace: 
https://jaeger.dev.storj.io/trace/131df032dcbcf2b
FAIL (2025-12-28 14:55:48 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.544131447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [..XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X.]
duration: 
9.173072819s
trace: 
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:55:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
 28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223    , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113    , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473   , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007   , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373   , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401   , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445   , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519   , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874   , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835   , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735     , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626     , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010     , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973     , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298     , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194   , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 9.173103637s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration: 
9.183801319s
trace: 
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:55:33 UTC) 
orbiter warning: context deadline exceeded
Duration: 9.183830975s edge gateway us-select-1