Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................XX............] duration: 4.162507325s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:30:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.162535327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................XX...........] duration: 3.673770189s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:30:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.673840521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 39.117853349s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:30:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 39.117883466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....] duration: 8.094760579s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 8.094789322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 32.380608726s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:29:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 32.380629977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................XX..........] duration: 3.539130749s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:29:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539161757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X....] duration: 6.245499181s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:28:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 6.245538026s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................XX.........] duration: 3.546031243s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:28:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546116355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................................X.........] duration: 15.412544967s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:28:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 15.412581067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................................X........] duration: 31.793915023s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 31.793942816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X...] duration: 5.883811085s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:27:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 5.883840342s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................XX........] duration: 3.569191338s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:27:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569214933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X..] duration: 6.030576761s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:26:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 6.030603412s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................XX.......] duration: 3.417075344s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:26:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417157018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................................X.......] duration: 32.779777834s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:26:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 32.779859088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................................X......] duration: 36.013650063s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:25:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 36.013673717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.] duration: 5.925510873s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:25:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 5.925535449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X] duration: 30.003879672s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 30.004038806s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................................XX......] duration: 3.511131938s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:25:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511162727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.................................................X.....] duration: 33.618626858s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:24:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 33.618644271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration: 3.516723748s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:24:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516749147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................................................X....] duration: 39.625084587s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:24:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 39.625110766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.] duration: 5.962505205s trace: https://jaeger.dev.storj.io/trace/29f5e30ea54e0e11 FAIL (2025-12-29 03:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:23:00.423908552 (563h54m0.294991587s) [upload=494406]: Starting upload Dec 29 03:23:00.423986470 (563h54m0.295069525s) [upload=494406]: Done starting upload Dec 29 03:23:00.424142898 (563h54m0.295225953s) [upload=494406]: Uploading segments... Dec 29 03:23:00.424265641 (563h54m0.295348676s) [upload=494406]: Got next segment. Inline: false Dec 29 03:23:00.424306720 (563h54m0.295389765s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:23:00.424312561 (563h54m0.295395606s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:23:00.666015594 (563h54m0.537098639s) [seg_pos={0 0}, upload=494406]: Begin upload segment... Dec 29 03:23:00.666090968 (563h54m0.537174003s) [seg_pos={0 0}, upload=494406]: Done begin upload segment. Dec 29 03:23:00.666161462 (563h54m0.537244507s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.666669039 (563h54m0.537752084s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667179370 (563h54m0.538262405s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667641770 (563h54m0.538724815s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668056700 (563h54m0.539139745s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668480868 (563h54m0.539563913s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668886040 (563h54m0.539969075s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.669359251 (563h54m0.540442286s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.670462291 (563h54m0.541545326s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.671732480 (563h54m0.542815515s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.672110159 (563h54m0.543193194s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.673556854 (563h54m0.544639899s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.676760584 (563h54m0.547843619s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.678919695 (563h54m0.55000273s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680330601 (563h54m0.551413636s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680833299 (563h54m0.551916334s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:02.290924598 (563h54m2.162007663s) [upload=494406]: Next returned nil segment Dec 29 03:23:02.290939015 (563h54m2.16202206s) [upload=494406]: Waiting for error group managing segments... Dec 29 03:23:02.670648359 (563h54m2.541731404s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.671044663 (563h54m2.542127708s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.680164559 (563h54m2.551247594s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.682448858 (563h54m2.553531903s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.683282345 (563h54m2.55436539s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686453123 (563h54m2.557536168s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686474604 (563h54m2.557557649s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.696583893 (563h54m2.567666939s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.722828235 (563h54m2.59391128s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.888880961 (563h54m2.759964006s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.954924927 (563h54m2.826007972s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.969743262 (563h54m2.840826307s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019432146 (563h54m2.890515191s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019461662 (563h54m2.890544707s) [seg_pos={0 0}, upload=494406]: Segment reached optimal threshold of 13 pieces. Dec 29 03:23:03.019745803 (563h54m2.890828848s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019759089 (563h54m2.890842164s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.019965993 (563h54m2.891049048s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019976102 (563h54m2.891059147s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020081803 (563h54m2.891164858s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.020103645 (563h54m2.89118669s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020135355 (563h54m2.89121839s) [seg_pos={0 0}, upload=494406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:23:03.020259231 (563h54m2.891342276s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:23:03.020266415 (563h54m2.89134945s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:23:03.304811753 (563h54m3.175894828s) [upload=494406]: Done uploading segments... Dec 29 03:23:03.304836951 (563h54m3.175920016s) [upload=494406]: Upload finished. err: <nil> Duration: 5.962531574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................XX....] duration: 5.001680382s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:23:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001697523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X] duration: 30.003016491s trace: https://jaeger.dev.storj.io/trace/29f5e30ea54e0e11 FAIL (2025-12-29 03:23:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:23:00.423908552 (563h54m0.294991587s) [upload=494406]: Starting upload Dec 29 03:23:00.423986470 (563h54m0.295069525s) [upload=494406]: Done starting upload Dec 29 03:23:00.424142898 (563h54m0.295225953s) [upload=494406]: Uploading segments... Dec 29 03:23:00.424265641 (563h54m0.295348676s) [upload=494406]: Got next segment. Inline: false Dec 29 03:23:00.424306720 (563h54m0.295389765s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:23:00.424312561 (563h54m0.295395606s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:23:00.666015594 (563h54m0.537098639s) [seg_pos={0 0}, upload=494406]: Begin upload segment... Dec 29 03:23:00.666090968 (563h54m0.537174003s) [seg_pos={0 0}, upload=494406]: Done begin upload segment. Dec 29 03:23:00.666161462 (563h54m0.537244507s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.666669039 (563h54m0.537752084s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667179370 (563h54m0.538262405s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667641770 (563h54m0.538724815s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668056700 (563h54m0.539139745s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668480868 (563h54m0.539563913s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668886040 (563h54m0.539969075s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.669359251 (563h54m0.540442286s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.670462291 (563h54m0.541545326s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.671732480 (563h54m0.542815515s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.672110159 (563h54m0.543193194s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.673556854 (563h54m0.544639899s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.676760584 (563h54m0.547843619s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.678919695 (563h54m0.55000273s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680330601 (563h54m0.551413636s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680833299 (563h54m0.551916334s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:02.290924598 (563h54m2.162007663s) [upload=494406]: Next returned nil segment Dec 29 03:23:02.290939015 (563h54m2.16202206s) [upload=494406]: Waiting for error group managing segments... Dec 29 03:23:02.670648359 (563h54m2.541731404s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.671044663 (563h54m2.542127708s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.680164559 (563h54m2.551247594s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.682448858 (563h54m2.553531903s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.683282345 (563h54m2.55436539s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686453123 (563h54m2.557536168s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686474604 (563h54m2.557557649s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.696583893 (563h54m2.567666939s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.722828235 (563h54m2.59391128s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.888880961 (563h54m2.759964006s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.954924927 (563h54m2.826007972s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.969743262 (563h54m2.840826307s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019432146 (563h54m2.890515191s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019461662 (563h54m2.890544707s) [seg_pos={0 0}, upload=494406]: Segment reached optimal threshold of 13 pieces. Dec 29 03:23:03.019745803 (563h54m2.890828848s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019759089 (563h54m2.890842164s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.019965993 (563h54m2.891049048s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019976102 (563h54m2.891059147s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020081803 (563h54m2.891164858s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.020103645 (563h54m2.89118669s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020135355 (563h54m2.89121839s) [seg_pos={0 0}, upload=494406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:23:03.020259231 (563h54m2.891342276s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:23:03.020266415 (563h54m2.89134945s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:23:03.304811753 (563h54m3.175894828s) [upload=494406]: Done uploading segments... Dec 29 03:23:03.304836951 (563h54m3.175920016s) [upload=494406]: Upload finished. err: <nil> Duration: 30.003036819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.................................................X...] duration: 33.443997154s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:23:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 33.444013905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................XX...] duration: 5.000886117s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:22:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000901966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................................................X..] duration: 40.067359977s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:22:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 40.067453324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................XX..] duration: 3.927847796s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:22:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.927861684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...] duration: 5.900911674s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 5.900945107s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 3.42510403s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:21:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425132766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.................................................X.] duration: 37.828731253s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:21:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 37.828797981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX..] duration: 5.585232019s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:21:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 5.585265344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................................................X] duration: 1m0.006178921s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 1m0.006192246s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.001861899s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:20:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001891254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX.] duration: 6.744566693s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:20:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 6.744608552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 29.217477881s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:20:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 29.217509653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX] duration: 30.004444593s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 30.004466864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001283474s trace: https://jaeger.dev.storj.io/trace/2d4f8cbe7bd1cb2 FAIL (2025-12-29 03:19:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001302631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.X] duration: 30.002613009s trace: https://jaeger.dev.storj.io/trace/7af3b20a9ea786c8 FAIL (2025-12-29 03:19:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== nts... Dec 29 03:18:48.323761711 (563h49m48.194844746s) [upload=494341]: Got next segment. Inline: false Dec 29 03:18:48.323787581 (563h49m48.194870616s) [upload=494341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:18:48.323791949 (563h49m48.194874984s) [upload=494341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:18:48.566262431 (563h49m48.437345476s) [seg_pos={0 0}, upload=494341]: Begin upload segment... Dec 29 03:18:48.566323838 (563h49m48.437406874s) [seg_pos={0 0}, upload=494341]: Done begin upload segment. Dec 29 03:18:48.566339728 (563h49m48.437422773s) [noise=true , address=145.79.227.52:4446 , piece_id=VG2QZ2BW5FOCLVNNF5LGEZVIDV7BJAHOQ4RKY5MQ6YDYEG2J7SIA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.566380806 (563h49m48.437463851s) [noise=true , address=145.79.227.15:4876 , piece_id=GSRWL6FEYAMNUWPBU2EQJ25YW3JGFO5MIJKWMWRALO4NTIQJRQJQ, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.566445460 (563h49m48.437528515s) [noise=true , address=145.79.227.36:4517 , piece_id=J74VJVCTWGFK2L74BBSJV4BYHS775JYALUG2AT2ITRHFDPCJNCOA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.566969778 (563h49m48.438052823s) [noise=true , address=145.79.227.30:4762 , piece_id=7W2422YEKX74UNB3O7SIA44V7Y3E7462HWKUWT6J67T5CB2RG7VA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.567339332 (563h49m48.438422367s) [noise=true , address=145.79.227.12:4767 , piece_id=2L5BIAMY4MRIZESXGORK7KIS5DJBZWFCLMZWBVSO3FU46T3IA6TA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.567671916 (563h49m48.438754951s) [noise=true , address=45.91.116.5:4298 , piece_id=XOK6KTG5EY576PYHNKS75GFIV5ISDDXFR6FOKEMIHKSU5UI6XJPQ, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.568030769 (563h49m48.439113814s) [noise=true , address=45.91.116.8:4048 , piece_id=CZJUZAHH3TF6IGJYRKXI6B4A4YW53JAF474ZI7YDY2U4CQDEPUKA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.568376126 (563h49m48.439459172s) [noise=true , address=145.79.227.50:4374 , piece_id=TLTWFRG2HGVWRW42TTWLUD5LBJSG525IYSXNQBMGE2G5QX65HEWA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.568741221 (563h49m48.439824256s) [noise=true , address=45.91.116.9:4261 , piece_id=PFJ6OGWJHRMHU2HDROK5UVA6TTHM2N6IUKGIOWJL425BHZINWW4Q, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.569148347 (563h49m48.440231392s) [noise=true , address=145.79.227.7:4293 , piece_id=ZRBHTGBPUXTW5FZKJZBLW72NEHYMJCXFKAZ5JKXPII47W7RKDVLA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.569615466 (563h49m48.440698501s) [noise=true , address=45.91.116.6:4975 , piece_id=FR46JPBMNJU5FS3YJBQAKAHIYAHACCANL7ZLYQZT37QWBJX7HBOA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.570085381 (563h49m48.441168416s) [noise=true , address=145.79.227.38:4979 , piece_id=PKRUMZQH523H2ZCKKRGKTAWKBGROKUBKKHTMQ5B556WV2HJXJBOA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.571250118 (563h49m48.442333154s) [noise=true , address=145.79.227.14:4837 , piece_id=MKFV6WEGTUQCLH43VVO7LR3B3ZP67VEQ4VOMGNQBLSO3ZOMFXQYQ, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.573280205 (563h49m48.44436324s) [noise=true , address=145.79.227.13:4802 , piece_id=XGHOKTIWILTLJXLSYJQV3NSKYHXYSOSE3XP3MGV4ERMTHNL7EURQ, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.575233975 (563h49m48.44631701s) [noise=true , address=45.91.116.10:4083 , piece_id=LZXDEGGL4QMEQRYESVXDHHNTPMXXOMK3BRH7QBB3V4BVI5F5GICQ, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:48.576889978 (563h49m48.447973023s) [noise=true , address=145.79.227.41:4087 , piece_id=VOFERI7LKBL73LFA57N2GIX5T34PK7Y2XOG4BAVRQS5HDOFSNAWA, seg_pos={0 0}, upload=494341]: Uploading piece... Dec 29 03:18:50.174284362 (563h49m50.045367397s) [upload=494341]: Next returned nil segment Dec 29 03:18:50.174299120 (563h49m50.045382165s) [upload=494341]: Waiting for error group managing segments... Dec 29 03:18:50.556467054 (563h49m50.427550099s) [noise=true , address=145.79.227.50:4374 , piece_id=TLTWFRG2HGVWRW42TTWLUD5LBJSG525IYSXNQBMGE2G5QX65HEWA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.557244564 (563h49m50.428327599s) [noise=true , address=145.79.227.15:4876 , piece_id=GSRWL6FEYAMNUWPBU2EQJ25YW3JGFO5MIJKWMWRALO4NTIQJRQJQ, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.559435506 (563h49m50.430518541s) [noise=true , address=145.79.227.36:4517 , piece_id=J74VJVCTWGFK2L74BBSJV4BYHS775JYALUG2AT2ITRHFDPCJNCOA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.564000147 (563h49m50.435083203s) [noise=true , address=145.79.227.13:4802 , piece_id=XGHOKTIWILTLJXLSYJQV3NSKYHXYSOSE3XP3MGV4ERMTHNL7EURQ, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.565850451 (563h49m50.436933486s) [noise=true , address=145.79.227.12:4767 , piece_id=2L5BIAMY4MRIZESXGORK7KIS5DJBZWFCLMZWBVSO3FU46T3IA6TA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.568005825 (563h49m50.43908887s) [noise=true , address=145.79.227.38:4979 , piece_id=PKRUMZQH523H2ZCKKRGKTAWKBGROKUBKKHTMQ5B556WV2HJXJBOA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.583149180 (563h49m50.454232256s) [noise=true , address=145.79.227.7:4293 , piece_id=ZRBHTGBPUXTW5FZKJZBLW72NEHYMJCXFKAZ5JKXPII47W7RKDVLA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.586014877 (563h49m50.457097932s) [noise=true , address=145.79.227.14:4837 , piece_id=MKFV6WEGTUQCLH43VVO7LR3B3ZP67VEQ4VOMGNQBLSO3ZOMFXQYQ, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.601345848 (563h49m50.472428893s) [noise=true , address=145.79.227.52:4446 , piece_id=VG2QZ2BW5FOCLVNNF5LGEZVIDV7BJAHOQ4RKY5MQ6YDYEG2J7SIA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.956105445 (563h49m50.82718848s) [noise=true , address=45.91.116.9:4261 , piece_id=PFJ6OGWJHRMHU2HDROK5UVA6TTHM2N6IUKGIOWJL425BHZINWW4Q, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.963694265 (563h49m50.8347773s) [noise=true , address=45.91.116.5:4298 , piece_id=XOK6KTG5EY576PYHNKS75GFIV5ISDDXFR6FOKEMIHKSU5UI6XJPQ, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:50.965570127 (563h49m50.836653172s) [noise=true , address=45.91.116.6:4975 , piece_id=FR46JPBMNJU5FS3YJBQAKAHIYAHACCANL7ZLYQZT37QWBJX7HBOA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:51.835672458 (563h49m51.706755493s) [noise=true , address=145.79.227.30:4762 , piece_id=7W2422YEKX74UNB3O7SIA44V7Y3E7462HWKUWT6J67T5CB2RG7VA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: <nil> Dec 29 03:18:51.835696353 (563h49m51.706779398s) [seg_pos={0 0}, upload=494341]: Segment reached optimal threshold of 13 pieces. Dec 29 03:18:51.835961298 (563h49m51.707044333s) [noise=true , address=145.79.227.41:4087 , piece_id=VOFERI7LKBL73LFA57N2GIX5T34PK7Y2XOG4BAVRQS5HDOFSNAWA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:36376->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:36376->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36376->145.79.227.41:4087: use of closed network connection Dec 29 03:18:51.835970615 (563h49m51.70705365s) [noise=true , address=145.79.227.41:4087 , piece_id=VOFERI7LKBL73LFA57N2GIX5T34PK7Y2XOG4BAVRQS5HDOFSNAWA, seg_pos={0 0}, upload=494341]: Long tail cancellation due to optimal threshold Dec 29 03:18:51.836060787 (563h49m51.707143823s) [noise=true , address=45.91.116.10:4083 , piece_id=LZXDEGGL4QMEQRYESVXDHHNTPMXXOMK3BRH7QBB3V4BVI5F5GICQ, seg_pos={0 0}, upload=494341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:57992->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:57992->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57992->45.91.116.10:4083: use of closed network connection Dec 29 03:18:51.836075305 (563h49m51.70715834s) [noise=true , address=45.91.116.10:4083 , piece_id=LZXDEGGL4QMEQRYESVXDHHNTPMXXOMK3BRH7QBB3V4BVI5F5GICQ, seg_pos={0 0}, upload=494341]: Long tail cancellation due to optimal threshold Dec 29 03:18:51.836196536 (563h49m51.707279581s) [noise=true , address=45.91.116.8:4048 , piece_id=CZJUZAHH3TF6IGJYRKXI6B4A4YW53JAF474ZI7YDY2U4CQDEPUKA, seg_pos={0 0}, upload=494341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 03:18:51.836213678 (563h49m51.707296713s) [noise=true , address=45.91.116.8:4048 , piece_id=CZJUZAHH3TF6IGJYRKXI6B4A4YW53JAF474ZI7YDY2U4CQDEPUKA, seg_pos={0 0}, upload=494341]: Long tail cancellation due to optimal threshold Dec 29 03:18:51.836233857 (563h49m51.707316902s) [seg_pos={0 0}, upload=494341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:18:51.836315141 (563h49m51.707398187s) [upload=494341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:18:51.836319460 (563h49m51.707402495s) [upload=494341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:18:52.127525151 (563h49m51.998608186s) [upload=494341]: Done uploading segments... Dec 29 03:18:52.127543726 (563h49m51.998626761s) [upload=494341]: Upload finished. err: <nil> Duration: 30.002636574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 41.795832717s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:19:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 41.795862063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.] duration: 11.136357037s trace: https://jaeger.dev.storj.io/trace/36264cfaad41c893 FAIL (2025-12-29 03:18:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: ZDMVJUAZYYTE5FJPGOC7Z4HRZNCT3OF6LRMCGYFI2QNYOZF25Q6Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZGJGG7R4QSG4OAZT7D25P73YVUKV3ESAP5KOPAJE5QIKRS4ZDLYQ) context deadline exceeded ======LOGS====== ments... Dec 29 03:17:13.233539881 (563h48m13.104622916s) [upload=494318]: Got next segment. Inline: false Dec 29 03:17:13.233569017 (563h48m13.104652052s) [upload=494318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:17:13.233574286 (563h48m13.104657321s) [upload=494318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:17:13.477974896 (563h48m13.349057931s) [seg_pos={0 0}, upload=494318]: Begin upload segment... Dec 29 03:17:13.478028407 (563h48m13.349111452s) [seg_pos={0 0}, upload=494318]: Done begin upload segment. Dec 29 03:17:13.478068244 (563h48m13.349151279s) [noise=true , address=145.79.227.37:4567 , piece_id=HZAMGEBFFMTAOTNPSC75GLR3JRPSESWEJM3DYBEC5BUO4NN3GUNA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.478099612 (563h48m13.349182647s) [noise=true , address=77.90.31.14:4664 , piece_id=ZGJGG7R4QSG4OAZT7D25P73YVUKV3ESAP5KOPAJE5QIKRS4ZDLYQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.478642056 (563h48m13.349725101s) [noise=true , address=145.79.227.15:4876 , piece_id=M5BCS2Z565FU2HCZ7XHD5KD6QV7LPCNQKE64ACFGV3UZOHJHNSAA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.479008785 (563h48m13.35009182s) [noise=true , address=145.79.227.50:4375 , piece_id=DCQNOEQPXPARLHP2Q2NLSA3TTNKCZJHEPQERDU5H7H4RJU4RHNPQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.479301562 (563h48m13.350384607s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDMVJUAZYYTE5FJPGOC7Z4HRZNCT3OF6LRMCGYFI2QNYOZF25Q6Q, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.480321053 (563h48m13.351404098s) [noise=true , address=145.79.227.42:4123 , piece_id=DUEFPQCFD65BVXNSTJ7BMH3HJV77ZD3OS52U37W7PSVHUEPIUZZQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.480693863 (563h48m13.351776898s) [noise=true , address=45.91.116.5:4298 , piece_id=NSFV3ZEZLX42JUEEVH4CBA3SXN2LODUFDGD2ODVDCQ6GODYGFUFQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.481869170 (563h48m13.352952205s) [noise=true , address=145.79.227.48:4303 , piece_id=PZ3LCX3ZCVIRXPDHLEBZ35T7NX4FJIBXYCWCGPA77X5LRKMEQ2BA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.482057028 (563h48m13.353140073s) [noise=true , address=145.79.227.32:4835 , piece_id=KW5VEQR2FN2JYCIHGMFNUAYGO6DA42J356YCXXP2WB5FUQDKUIAQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.483496088 (563h48m13.354579133s) [noise=true , address=145.79.227.43:4159 , piece_id=5MLWCRMA6VPENPYYT3UWZEJFPPMQUYIKDMAAE6W2U3E4ROO4TMTA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.485001094 (563h48m13.356084129s) [noise=true , address=145.79.227.2:4113 , piece_id=BQ5ZIYHJ5SPDSYVRVUAZ4BMWM4GCEPFNXC2E4UAUZEWYWYYG7TCA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.486646748 (563h48m13.357729783s) [noise=true , address=145.79.227.13:4802 , piece_id=QEBUV2T3QYZ3DIAAXGUPVTPENKASWBMQGJVFSC5KIBBURBHFMFAA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.487965448 (563h48m13.359048483s) [noise=true , address=145.79.227.47:4267 , piece_id=TH5C5JOHLNJUQVWHTNQTYVIVIT4YTYEHHFIV7AG4MIVU3K5NMUTA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.488287301 (563h48m13.359370336s) [noise=true , address=77.90.31.10:4519 , piece_id=OD6PK5Q2R2ZYCII4ANC2DHY5DV7XZWW27XXZZG3ZDAIC7L2SGDQQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.489563712 (563h48m13.360646747s) [noise=true , address=145.79.227.41:4085 , piece_id=ZMMHHL3EVAR5GSBB5BKSO4KYXWHHOZARMPBWELJRBET6EZ4IOJIA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.490278342 (563h48m13.361361377s) [noise=true , address=145.79.227.21:4473 , piece_id=PKJAPW5OCUD46AD67CKA3MQVMBNSMKXFDNQJ55PHIAPORLOQ662A, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:15.177600331 (563h48m15.048683366s) [upload=494318]: Next returned nil segment Dec 29 03:17:15.177613447 (563h48m15.048696492s) [upload=494318]: Waiting for error group managing segments... Dec 29 03:17:15.459173882 (563h48m15.330256917s) [noise=true , address=145.79.227.15:4876 , piece_id=M5BCS2Z565FU2HCZ7XHD5KD6QV7LPCNQKE64ACFGV3UZOHJHNSAA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.459256599 (563h48m15.330339644s) [noise=true , address=145.79.227.41:4085 , piece_id=ZMMHHL3EVAR5GSBB5BKSO4KYXWHHOZARMPBWELJRBET6EZ4IOJIA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.459814031 (563h48m15.330897076s) [noise=true , address=145.79.227.48:4303 , piece_id=PZ3LCX3ZCVIRXPDHLEBZ35T7NX4FJIBXYCWCGPA77X5LRKMEQ2BA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.468867139 (563h48m15.339950174s) [noise=true , address=145.79.227.47:4267 , piece_id=TH5C5JOHLNJUQVWHTNQTYVIVIT4YTYEHHFIV7AG4MIVU3K5NMUTA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.468957031 (563h48m15.340040066s) [noise=true , address=145.79.227.37:4567 , piece_id=HZAMGEBFFMTAOTNPSC75GLR3JRPSESWEJM3DYBEC5BUO4NN3GUNA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.472507791 (563h48m15.343590826s) [noise=true , address=145.79.227.13:4802 , piece_id=QEBUV2T3QYZ3DIAAXGUPVTPENKASWBMQGJVFSC5KIBBURBHFMFAA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.477178725 (563h48m15.34826176s) [noise=true , address=145.79.227.21:4473 , piece_id=PKJAPW5OCUD46AD67CKA3MQVMBNSMKXFDNQJ55PHIAPORLOQ662A, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.482668669 (563h48m15.353751704s) [noise=true , address=145.79.227.50:4375 , piece_id=DCQNOEQPXPARLHP2Q2NLSA3TTNKCZJHEPQERDU5H7H4RJU4RHNPQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.493027223 (563h48m15.364110258s) [noise=true , address=145.79.227.32:4835 , piece_id=KW5VEQR2FN2JYCIHGMFNUAYGO6DA42J356YCXXP2WB5FUQDKUIAQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.495092275 (563h48m15.36617531s) [noise=true , address=145.79.227.42:4123 , piece_id=DUEFPQCFD65BVXNSTJ7BMH3HJV77ZD3OS52U37W7PSVHUEPIUZZQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.661066119 (563h48m15.532149154s) [noise=true , address=77.90.31.14:4664 , piece_id=ZGJGG7R4QSG4OAZT7D25P73YVUKV3ESAP5KOPAJE5QIKRS4ZDLYQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.726419099 (563h48m15.597502134s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDMVJUAZYYTE5FJPGOC7Z4HRZNCT3OF6LRMCGYFI2QNYOZF25Q6Q, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.829565903 (563h48m15.700648938s) [noise=true , address=45.91.116.5:4298 , piece_id=NSFV3ZEZLX42JUEEVH4CBA3SXN2LODUFDGD2ODVDCQ6GODYGFUFQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.829591693 (563h48m15.700674728s) [seg_pos={0 0}, upload=494318]: Segment reached optimal threshold of 13 pieces. Dec 29 03:17:15.829877116 (563h48m15.700960171s) [noise=true , address=77.90.31.10:4519 , piece_id=OD6PK5Q2R2ZYCII4ANC2DHY5DV7XZWW27XXZZG3ZDAIC7L2SGDQQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56508->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56508->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56508->77.90.31.10:4519: use of closed network connection Dec 29 03:17:15.829886715 (563h48m15.70096975s) [noise=true , address=77.90.31.10:4519 , piece_id=OD6PK5Q2R2ZYCII4ANC2DHY5DV7XZWW27XXZZG3ZDAIC7L2SGDQQ, seg_pos={0 0}, upload=494318]: Long tail cancellation due to optimal threshold Dec 29 03:17:15.830062570 (563h48m15.701145615s) [noise=true , address=145.79.227.2:4113 , piece_id=BQ5ZIYHJ5SPDSYVRVUAZ4BMWM4GCEPFNXC2E4UAUZEWYWYYG7TCA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:60614->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:60614->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60614->145.79.227.2:4113: use of closed network connection Dec 29 03:17:15.830100883 (563h48m15.701183918s) [noise=true , address=145.79.227.2:4113 , piece_id=BQ5ZIYHJ5SPDSYVRVUAZ4BMWM4GCEPFNXC2E4UAUZEWYWYYG7TCA, seg_pos={0 0}, upload=494318]: Long tail cancellation due to optimal threshold Dec 29 03:17:15.830302616 (563h48m15.701385651s) [noise=true , address=145.79.227.43:4159 , piece_id=5MLWCRMA6VPENPYYT3UWZEJFPPMQUYIKDMAAE6W2U3E4ROO4TMTA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 03:17:15.830310873 (563h48m15.701393918s) [noise=true , address=145.79.227.43:4159 , piece_id=5MLWCRMA6VPENPYYT3UWZEJFPPMQUYIKDMAAE6W2U3E4ROO4TMTA, seg_pos={0 0}, upload=494318]: Long tail cancellation due to optimal threshold Dec 29 03:17:15.830330920 (563h48m15.701413955s) [seg_pos={0 0}, upload=494318]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:17:15.830434808 (563h48m15.701517853s) [upload=494318]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:17:15.830441150 (563h48m15.701524195s) [upload=494318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:17:16.118647721 (563h48m15.989730766s) [upload=494318]: Done uploading segments... Dec 29 03:17:16.118674412 (563h48m15.989757457s) [upload=494318]: Upload finished. err: <nil> Duration: 11.136388316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 23.837389319s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 23.837408285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX..X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX] duration: 30.001954135s trace: https://jaeger.dev.storj.io/trace/36264cfaad41c893 FAIL (2025-12-29 03:17:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: ZDMVJUAZYYTE5FJPGOC7Z4HRZNCT3OF6LRMCGYFI2QNYOZF25Q6Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZGJGG7R4QSG4OAZT7D25P73YVUKV3ESAP5KOPAJE5QIKRS4ZDLYQ) context deadline exceeded ======LOGS====== ments... Dec 29 03:17:13.233539881 (563h48m13.104622916s) [upload=494318]: Got next segment. Inline: false Dec 29 03:17:13.233569017 (563h48m13.104652052s) [upload=494318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:17:13.233574286 (563h48m13.104657321s) [upload=494318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:17:13.477974896 (563h48m13.349057931s) [seg_pos={0 0}, upload=494318]: Begin upload segment... Dec 29 03:17:13.478028407 (563h48m13.349111452s) [seg_pos={0 0}, upload=494318]: Done begin upload segment. Dec 29 03:17:13.478068244 (563h48m13.349151279s) [noise=true , address=145.79.227.37:4567 , piece_id=HZAMGEBFFMTAOTNPSC75GLR3JRPSESWEJM3DYBEC5BUO4NN3GUNA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.478099612 (563h48m13.349182647s) [noise=true , address=77.90.31.14:4664 , piece_id=ZGJGG7R4QSG4OAZT7D25P73YVUKV3ESAP5KOPAJE5QIKRS4ZDLYQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.478642056 (563h48m13.349725101s) [noise=true , address=145.79.227.15:4876 , piece_id=M5BCS2Z565FU2HCZ7XHD5KD6QV7LPCNQKE64ACFGV3UZOHJHNSAA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.479008785 (563h48m13.35009182s) [noise=true , address=145.79.227.50:4375 , piece_id=DCQNOEQPXPARLHP2Q2NLSA3TTNKCZJHEPQERDU5H7H4RJU4RHNPQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.479301562 (563h48m13.350384607s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDMVJUAZYYTE5FJPGOC7Z4HRZNCT3OF6LRMCGYFI2QNYOZF25Q6Q, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.480321053 (563h48m13.351404098s) [noise=true , address=145.79.227.42:4123 , piece_id=DUEFPQCFD65BVXNSTJ7BMH3HJV77ZD3OS52U37W7PSVHUEPIUZZQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.480693863 (563h48m13.351776898s) [noise=true , address=45.91.116.5:4298 , piece_id=NSFV3ZEZLX42JUEEVH4CBA3SXN2LODUFDGD2ODVDCQ6GODYGFUFQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.481869170 (563h48m13.352952205s) [noise=true , address=145.79.227.48:4303 , piece_id=PZ3LCX3ZCVIRXPDHLEBZ35T7NX4FJIBXYCWCGPA77X5LRKMEQ2BA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.482057028 (563h48m13.353140073s) [noise=true , address=145.79.227.32:4835 , piece_id=KW5VEQR2FN2JYCIHGMFNUAYGO6DA42J356YCXXP2WB5FUQDKUIAQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.483496088 (563h48m13.354579133s) [noise=true , address=145.79.227.43:4159 , piece_id=5MLWCRMA6VPENPYYT3UWZEJFPPMQUYIKDMAAE6W2U3E4ROO4TMTA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.485001094 (563h48m13.356084129s) [noise=true , address=145.79.227.2:4113 , piece_id=BQ5ZIYHJ5SPDSYVRVUAZ4BMWM4GCEPFNXC2E4UAUZEWYWYYG7TCA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.486646748 (563h48m13.357729783s) [noise=true , address=145.79.227.13:4802 , piece_id=QEBUV2T3QYZ3DIAAXGUPVTPENKASWBMQGJVFSC5KIBBURBHFMFAA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.487965448 (563h48m13.359048483s) [noise=true , address=145.79.227.47:4267 , piece_id=TH5C5JOHLNJUQVWHTNQTYVIVIT4YTYEHHFIV7AG4MIVU3K5NMUTA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.488287301 (563h48m13.359370336s) [noise=true , address=77.90.31.10:4519 , piece_id=OD6PK5Q2R2ZYCII4ANC2DHY5DV7XZWW27XXZZG3ZDAIC7L2SGDQQ, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.489563712 (563h48m13.360646747s) [noise=true , address=145.79.227.41:4085 , piece_id=ZMMHHL3EVAR5GSBB5BKSO4KYXWHHOZARMPBWELJRBET6EZ4IOJIA, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:13.490278342 (563h48m13.361361377s) [noise=true , address=145.79.227.21:4473 , piece_id=PKJAPW5OCUD46AD67CKA3MQVMBNSMKXFDNQJ55PHIAPORLOQ662A, seg_pos={0 0}, upload=494318]: Uploading piece... Dec 29 03:17:15.177600331 (563h48m15.048683366s) [upload=494318]: Next returned nil segment Dec 29 03:17:15.177613447 (563h48m15.048696492s) [upload=494318]: Waiting for error group managing segments... Dec 29 03:17:15.459173882 (563h48m15.330256917s) [noise=true , address=145.79.227.15:4876 , piece_id=M5BCS2Z565FU2HCZ7XHD5KD6QV7LPCNQKE64ACFGV3UZOHJHNSAA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.459256599 (563h48m15.330339644s) [noise=true , address=145.79.227.41:4085 , piece_id=ZMMHHL3EVAR5GSBB5BKSO4KYXWHHOZARMPBWELJRBET6EZ4IOJIA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.459814031 (563h48m15.330897076s) [noise=true , address=145.79.227.48:4303 , piece_id=PZ3LCX3ZCVIRXPDHLEBZ35T7NX4FJIBXYCWCGPA77X5LRKMEQ2BA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.468867139 (563h48m15.339950174s) [noise=true , address=145.79.227.47:4267 , piece_id=TH5C5JOHLNJUQVWHTNQTYVIVIT4YTYEHHFIV7AG4MIVU3K5NMUTA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.468957031 (563h48m15.340040066s) [noise=true , address=145.79.227.37:4567 , piece_id=HZAMGEBFFMTAOTNPSC75GLR3JRPSESWEJM3DYBEC5BUO4NN3GUNA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.472507791 (563h48m15.343590826s) [noise=true , address=145.79.227.13:4802 , piece_id=QEBUV2T3QYZ3DIAAXGUPVTPENKASWBMQGJVFSC5KIBBURBHFMFAA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.477178725 (563h48m15.34826176s) [noise=true , address=145.79.227.21:4473 , piece_id=PKJAPW5OCUD46AD67CKA3MQVMBNSMKXFDNQJ55PHIAPORLOQ662A, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.482668669 (563h48m15.353751704s) [noise=true , address=145.79.227.50:4375 , piece_id=DCQNOEQPXPARLHP2Q2NLSA3TTNKCZJHEPQERDU5H7H4RJU4RHNPQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.493027223 (563h48m15.364110258s) [noise=true , address=145.79.227.32:4835 , piece_id=KW5VEQR2FN2JYCIHGMFNUAYGO6DA42J356YCXXP2WB5FUQDKUIAQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.495092275 (563h48m15.36617531s) [noise=true , address=145.79.227.42:4123 , piece_id=DUEFPQCFD65BVXNSTJ7BMH3HJV77ZD3OS52U37W7PSVHUEPIUZZQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.661066119 (563h48m15.532149154s) [noise=true , address=77.90.31.14:4664 , piece_id=ZGJGG7R4QSG4OAZT7D25P73YVUKV3ESAP5KOPAJE5QIKRS4ZDLYQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.726419099 (563h48m15.597502134s) [noise=true , address=45.91.116.12:4225 , piece_id=ZDMVJUAZYYTE5FJPGOC7Z4HRZNCT3OF6LRMCGYFI2QNYOZF25Q6Q, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.829565903 (563h48m15.700648938s) [noise=true , address=45.91.116.5:4298 , piece_id=NSFV3ZEZLX42JUEEVH4CBA3SXN2LODUFDGD2ODVDCQ6GODYGFUFQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: <nil> Dec 29 03:17:15.829591693 (563h48m15.700674728s) [seg_pos={0 0}, upload=494318]: Segment reached optimal threshold of 13 pieces. Dec 29 03:17:15.829877116 (563h48m15.700960171s) [noise=true , address=77.90.31.10:4519 , piece_id=OD6PK5Q2R2ZYCII4ANC2DHY5DV7XZWW27XXZZG3ZDAIC7L2SGDQQ, seg_pos={0 0}, upload=494318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56508->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56508->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56508->77.90.31.10:4519: use of closed network connection Dec 29 03:17:15.829886715 (563h48m15.70096975s) [noise=true , address=77.90.31.10:4519 , piece_id=OD6PK5Q2R2ZYCII4ANC2DHY5DV7XZWW27XXZZG3ZDAIC7L2SGDQQ, seg_pos={0 0}, upload=494318]: Long tail cancellation due to optimal threshold Dec 29 03:17:15.830062570 (563h48m15.701145615s) [noise=true , address=145.79.227.2:4113 , piece_id=BQ5ZIYHJ5SPDSYVRVUAZ4BMWM4GCEPFNXC2E4UAUZEWYWYYG7TCA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:60614->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:60614->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60614->145.79.227.2:4113: use of closed network connection Dec 29 03:17:15.830100883 (563h48m15.701183918s) [noise=true , address=145.79.227.2:4113 , piece_id=BQ5ZIYHJ5SPDSYVRVUAZ4BMWM4GCEPFNXC2E4UAUZEWYWYYG7TCA, seg_pos={0 0}, upload=494318]: Long tail cancellation due to optimal threshold Dec 29 03:17:15.830302616 (563h48m15.701385651s) [noise=true , address=145.79.227.43:4159 , piece_id=5MLWCRMA6VPENPYYT3UWZEJFPPMQUYIKDMAAE6W2U3E4ROO4TMTA, seg_pos={0 0}, upload=494318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 03:17:15.830310873 (563h48m15.701393918s) [noise=true , address=145.79.227.43:4159 , piece_id=5MLWCRMA6VPENPYYT3UWZEJFPPMQUYIKDMAAE6W2U3E4ROO4TMTA, seg_pos={0 0}, upload=494318]: Long tail cancellation due to optimal threshold Dec 29 03:17:15.830330920 (563h48m15.701413955s) [seg_pos={0 0}, upload=494318]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:17:15.830434808 (563h48m15.701517853s) [upload=494318]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:17:15.830441150 (563h48m15.701524195s) [upload=494318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:17:16.118647721 (563h48m15.989730766s) [upload=494318]: Done uploading segments... Dec 29 03:17:16.118674412 (563h48m15.989757457s) [upload=494318]: Upload finished. err: <nil> Duration: 30.001976848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 25.350091339s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:17:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 25.350112258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 27.106106194s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:17:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 27.106123466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 40.24482118s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:16:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 40.244837831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXX..X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XX] duration: 30.002862971s trace: https://jaeger.dev.storj.io/trace/3d3f7d299f189acc FAIL (2025-12-29 03:16:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: TS4SFHR76Z2BJYSL2DNYFCBAVXKOQXHRZ5YTCPG2LKZ4WAHSF4QQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GXGS5PAA2TCEFLWXUAEE5UZ4X7X6MNXHBZQFI5EZPYVS6A2KKCYA) context deadline exceeded ======LOGS====== Dec 29 03:16:00.939722954 (563h47m0.810805989s) [upload=494299]: Starting upload Dec 29 03:16:00.939819638 (563h47m0.810902683s) [upload=494299]: Done starting upload Dec 29 03:16:00.939872739 (563h47m0.810955794s) [upload=494299]: Uploading segments... Dec 29 03:16:00.940079402 (563h47m0.811162447s) [upload=494299]: Got next segment. Inline: false Dec 29 03:16:00.940144707 (563h47m0.811227752s) [upload=494299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:16:00.940150618 (563h47m0.811233663s) [upload=494299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:16:01.202121964 (563h47m1.073205019s) [seg_pos={0 0}, upload=494299]: Begin upload segment... Dec 29 03:16:01.202176608 (563h47m1.073259643s) [seg_pos={0 0}, upload=494299]: Done begin upload segment. Dec 29 03:16:01.202204240 (563h47m1.073287305s) [noise=true , address=145.79.227.12:4767 , piece_id=G45GASODJFSBBDEYBKPRNRNM22PBP3LON7EVBVLKMKAXBGT6DJZA, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.202219669 (563h47m1.073302714s) [noise=true , address=145.79.227.29:4943 , piece_id=OQCHTAVWXACFCX7GSMHD4JNSXSTMTGRH7ZER6BB4SVY2K4G2T25Q, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.202281106 (563h47m1.073364152s) [noise=true , address=145.79.227.14:4840 , piece_id=OINZ3KRXO3H6TXKEO5I65BKCSE3NMZYOVQ7CUFVOIDUJDYXGRGVQ, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.203097141 (563h47m1.074180196s) [noise=true , address=45.91.116.13:4155 , piece_id=TS4SFHR76Z2BJYSL2DNYFCBAVXKOQXHRZ5YTCPG2LKZ4WAHSF4QQ, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.203574429 (563h47m1.074657474s) [noise=true , address=145.79.227.5:4221 , piece_id=ODQB3XIKDGL77DHO6QHOVSHM2YH2H4GWVRIUVLL3HV3C2YK4GCJQ, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.204017192 (563h47m1.075100238s) [noise=true , address=145.79.227.48:4302 , piece_id=JKQQ33K4RSULSYDIS5CHJBGOJED6H4YGYWQQYP6ZVCAW2XAML7QA, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.204520340 (563h47m1.075603385s) [noise=true , address=145.79.227.46:4602 , piece_id=4NZPFDNQSZYHS5EHIN3OBNEPQMV6KYZ4ON4ST746CUBSCUQAOS4A, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.205015363 (563h47m1.076098409s) [noise=true , address=145.79.227.13:4803 , piece_id=FS6OBCIJ4743LQGWDCUTH7TJZUXBI2K23V6JCDJFFOICC4UH4RYA, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.205637067 (563h47m1.076720112s) [noise=true , address=77.90.31.21:4877 , piece_id=IBJ6VY5WZNMGV3CBY7IBV4IFPRSXBRBJPHG224XJ5U26INDUBBNA, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.206513045 (563h47m1.07759609s) [noise=true , address=45.91.116.9:4262 , piece_id=QGMKJXNK4KWP2OUAWEVKSS63GI3IYVOAUE6XQYWMMIC2K6AJ4HSQ, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.207121073 (563h47m1.078204118s) [noise=true , address=45.91.116.6:4975 , piece_id=5IVZUVFQHNRUU6Z3RMMOCXBCKK3USNELWF2RDBP24GNSAZR2LK2Q, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.207667043 (563h47m1.078750078s) [noise=true , address=145.79.227.11:4729 , piece_id=4FDL5QJCJFW4TIQFPSZP2GZBF54AWIOUA4J4DFXFFMIC2QV3VCDA, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.209099199 (563h47m1.080182245s) [noise=true , address=45.91.116.8:4046 , piece_id=GXGS5PAA2TCEFLWXUAEE5UZ4X7X6MNXHBZQFI5EZPYVS6A2KKCYA, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.210992274 (563h47m1.082075319s) [noise=true , address=45.91.116.11:4117 , piece_id=KEAFJF7MV3MAWU7PLNOB22C3GZFSKEXYTWQZIJE7NJEJSJI67DBQ, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.212929573 (563h47m1.084012608s) [noise=true , address=145.79.227.38:4979 , piece_id=NG7LLBIVZ2OJ6KENYK2XM567NRDTLVMXR6X2WIZLC43SRP7JNZHQ, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:01.215332759 (563h47m1.086415804s) [noise=true , address=145.79.227.26:4653 , piece_id=IBKLO6P5HQEVSERPJ4OPGMVRG2PKTRTM4TM6SUT2AD4A7ZRIQA5Q, seg_pos={0 0}, upload=494299]: Uploading piece... Dec 29 03:16:02.917697862 (563h47m2.788780907s) [upload=494299]: Next returned nil segment Dec 29 03:16:02.917711808 (563h47m2.788794853s) [upload=494299]: Waiting for error group managing segments... Dec 29 03:16:03.200061337 (563h47m3.071144372s) [noise=true , address=145.79.227.48:4302 , piece_id=JKQQ33K4RSULSYDIS5CHJBGOJED6H4YGYWQQYP6ZVCAW2XAML7QA, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.208440252 (563h47m3.079523297s) [noise=true , address=145.79.227.13:4803 , piece_id=FS6OBCIJ4743LQGWDCUTH7TJZUXBI2K23V6JCDJFFOICC4UH4RYA, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.211715408 (563h47m3.082798453s) [noise=true , address=145.79.227.12:4767 , piece_id=G45GASODJFSBBDEYBKPRNRNM22PBP3LON7EVBVLKMKAXBGT6DJZA, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.212342071 (563h47m3.083425116s) [noise=true , address=145.79.227.14:4840 , piece_id=OINZ3KRXO3H6TXKEO5I65BKCSE3NMZYOVQ7CUFVOIDUJDYXGRGVQ, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.212880937 (563h47m3.083963982s) [noise=true , address=145.79.227.5:4221 , piece_id=ODQB3XIKDGL77DHO6QHOVSHM2YH2H4GWVRIUVLL3HV3C2YK4GCJQ, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.219006060 (563h47m3.090089105s) [noise=true , address=145.79.227.46:4602 , piece_id=4NZPFDNQSZYHS5EHIN3OBNEPQMV6KYZ4ON4ST746CUBSCUQAOS4A, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.223396459 (563h47m3.094479504s) [noise=true , address=145.79.227.11:4729 , piece_id=4FDL5QJCJFW4TIQFPSZP2GZBF54AWIOUA4J4DFXFFMIC2QV3VCDA, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.230300446 (563h47m3.101383491s) [noise=true , address=145.79.227.26:4653 , piece_id=IBKLO6P5HQEVSERPJ4OPGMVRG2PKTRTM4TM6SUT2AD4A7ZRIQA5Q, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.231939998 (563h47m3.103023053s) [noise=true , address=145.79.227.38:4979 , piece_id=NG7LLBIVZ2OJ6KENYK2XM567NRDTLVMXR6X2WIZLC43SRP7JNZHQ, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.378284505 (563h47m3.24936755s) [noise=true , address=77.90.31.21:4877 , piece_id=IBJ6VY5WZNMGV3CBY7IBV4IFPRSXBRBJPHG224XJ5U26INDUBBNA, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.527534647 (563h47m3.398617692s) [noise=true , address=45.91.116.11:4117 , piece_id=KEAFJF7MV3MAWU7PLNOB22C3GZFSKEXYTWQZIJE7NJEJSJI67DBQ, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:03.582230604 (563h47m3.453313649s) [noise=true , address=45.91.116.13:4155 , piece_id=TS4SFHR76Z2BJYSL2DNYFCBAVXKOQXHRZ5YTCPG2LKZ4WAHSF4QQ, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:04.350885108 (563h47m4.221968153s) [noise=true , address=45.91.116.8:4046 , piece_id=GXGS5PAA2TCEFLWXUAEE5UZ4X7X6MNXHBZQFI5EZPYVS6A2KKCYA, seg_pos={0 0}, upload=494299]: Done uploading piece. err: <nil> Dec 29 03:16:04.350920596 (563h47m4.222003641s) [seg_pos={0 0}, upload=494299]: Segment reached optimal threshold of 13 pieces. Dec 29 03:16:04.351222631 (563h47m4.222305676s) [noise=true , address=45.91.116.9:4262 , piece_id=QGMKJXNK4KWP2OUAWEVKSS63GI3IYVOAUE6XQYWMMIC2K6AJ4HSQ, seg_pos={0 0}, upload=494299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 03:16:04.351244543 (563h47m4.222327578s) [noise=true , address=45.91.116.9:4262 , piece_id=QGMKJXNK4KWP2OUAWEVKSS63GI3IYVOAUE6XQYWMMIC2K6AJ4HSQ, seg_pos={0 0}, upload=494299]: Long tail cancellation due to optimal threshold Dec 29 03:16:04.351263980 (563h47m4.222347025s) [noise=true , address=45.91.116.6:4975 , piece_id=5IVZUVFQHNRUU6Z3RMMOCXBCKK3USNELWF2RDBP24GNSAZR2LK2Q, seg_pos={0 0}, upload=494299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42786->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 03:16:04.351273108 (563h47m4.222356153s) [noise=true , address=45.91.116.6:4975 , piece_id=5IVZUVFQHNRUU6Z3RMMOCXBCKK3USNELWF2RDBP24GNSAZR2LK2Q, seg_pos={0 0}, upload=494299]: Long tail cancellation due to optimal threshold Dec 29 03:16:04.351461286 (563h47m4.222544331s) [noise=true , address=145.79.227.29:4943 , piece_id=OQCHTAVWXACFCX7GSMHD4JNSXSTMTGRH7ZER6BB4SVY2K4G2T25Q, seg_pos={0 0}, upload=494299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 03:16:04.351470192 (563h47m4.222553238s) [noise=true , address=145.79.227.29:4943 , piece_id=OQCHTAVWXACFCX7GSMHD4JNSXSTMTGRH7ZER6BB4SVY2K4G2T25Q, seg_pos={0 0}, upload=494299]: Long tail cancellation due to optimal threshold Dec 29 03:16:04.351490502 (563h47m4.222573547s) [seg_pos={0 0}, upload=494299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:16:04.351575954 (563h47m4.222658999s) [upload=494299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:16:04.351581444 (563h47m4.222664489s) [upload=494299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:16:04.646784549 (563h47m4.517867594s) [upload=494299]: Done uploading segments... Dec 29 03:16:04.646811379 (563h47m4.517894424s) [upload=494299]: Upload finished. err: <nil> Duration: 30.002891423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 32.629219045s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:15:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 32.629246848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XXX..X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..X] duration: 30.004904186s trace: https://jaeger.dev.storj.io/trace/250d887ec875d648 FAIL (2025-12-29 03:15:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: E2ITHUYLLMXKSIMXIFELWUQ6NKBPT2YA74H4KB4BCYJ7FHIKU6DQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: L7ZF6WDJIO6RN7UVN6DIXK5OCANBUPEM3UZI4UWFABXGHS7T73TQ) context deadline exceeded ======LOGS====== 14:47.643937017 (563h45m47.515020052s) [upload=494284]: Starting upload Dec 29 03:14:47.643986621 (563h45m47.515069656s) [upload=494284]: Done starting upload Dec 29 03:14:47.644157266 (563h45m47.515240301s) [upload=494284]: Uploading segments... Dec 29 03:14:47.644193635 (563h45m47.51527667s) [upload=494284]: Got next segment. Inline: false Dec 29 03:14:47.644217231 (563h45m47.515300266s) [upload=494284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:14:47.644221208 (563h45m47.515304243s) [upload=494284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:14:47.888633750 (563h45m47.759716785s) [seg_pos={0 0}, upload=494284]: Begin upload segment... Dec 29 03:14:47.888683585 (563h45m47.75976662s) [seg_pos={0 0}, upload=494284]: Done begin upload segment. Dec 29 03:14:47.888776802 (563h45m47.759859837s) [noise=true , address=145.79.227.47:4265 , piece_id=MVL5FDS6ATIWW5QZSKI27Y7LUHISG2ZHPBQ766U4T7Y2UEK6YLWA, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.888782413 (563h45m47.759865448s) [noise=true , address=145.79.227.13:4802 , piece_id=XEND4N4V3WV6VQZ2CS3BMLYR4JTVFGNKBBO7DBCK3ROZW7J56ZYA, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.889537090 (563h45m47.760620125s) [noise=true , address=45.91.116.7:4011 , piece_id=56B5LJUXG4JABE4MNI6T4LBKGCC6ATLCZTVBXRUC2OFLKFZWV5AQ, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.890007907 (563h45m47.761090952s) [noise=true , address=145.79.227.52:4445 , piece_id=X3QXKP5UAOM3VT6HHPG2MB6AZATTGFXJNYZFKV52IQ5HFOHOW3HA, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.890083290 (563h45m47.761166325s) [noise=true , address=77.90.31.19:4842 , piece_id=UKKYNA5BG3P24MHZSGMZWLOCHPHOEKNAI23LNZ7OJAUUNDLD62QQ, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.890410062 (563h45m47.761493117s) [noise=true , address=145.79.227.2:4113 , piece_id=L4IIMJFDKZA2P77SL5EFAC4NQGOKPUBY3CP6AFJOMS5N4EFP4GTQ, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.890636414 (563h45m47.761719449s) [noise=true , address=145.79.227.48:4302 , piece_id=WHHBMYZGM4E6J53GZ4D4WX3F62GTN7BSGXBTJJLV3XOWMWZYLAJA, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.890890036 (563h45m47.761973071s) [noise=true , address=145.79.227.11:4729 , piece_id=V3JTWDCUENFAZ46KFCSJIWFMP6WL73JSNZU2CJEPNWKZMHRNPY5A, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.891177263 (563h45m47.762260298s) [noise=true , address=45.91.116.6:4976 , piece_id=L7ZF6WDJIO6RN7UVN6DIXK5OCANBUPEM3UZI4UWFABXGHS7T73TQ, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.891692594 (563h45m47.762775629s) [noise=true , address=145.79.227.36:4517 , piece_id=4TRR2Y2IWTXG4IHH3VE4Z2MGH6MF3CN4BUFHOYJPZMRAOFRQVA2A, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.892121801 (563h45m47.763204836s) [noise=true , address=45.91.116.12:4225 , piece_id=Y22G6GFWCXVOQM7RZCHTILKK52XXDJL2GEK2JJLIX5REWNQIGXUA, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.892462661 (563h45m47.763545696s) [noise=true , address=77.90.31.10:4518 , piece_id=E2ITHUYLLMXKSIMXIFELWUQ6NKBPT2YA74H4KB4BCYJ7FHIKU6DQ, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.892981559 (563h45m47.764064594s) [noise=true , address=145.79.227.15:4873 , piece_id=XM65EB6TF6YVZGWRA4DATYYLRYJMJZHM6C6XAJLFH27ZBZTOPX4A, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.893362324 (563h45m47.764445359s) [noise=true , address=77.90.31.11:4555 , piece_id=5CLAKU6F2TES7YHTUG5ZM4FKYGDATEEXZZGAFJTEDL5KDBFTKKNA, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.894940288 (563h45m47.766023323s) [noise=true , address=145.79.227.19:4077 , piece_id=DIG35FEKP2N5H4UXGNJPY6S4NLLSKY3XLZIVZIQXMG7R4JXPDSDQ, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:47.895322266 (563h45m47.766405311s) [noise=true , address=145.79.227.29:4941 , piece_id=XT3NRLFCWGODRZNNTFSRJQYJ2WGQSZRNVKF6YT3IKE2YMUULAC7A, seg_pos={0 0}, upload=494284]: Uploading piece... Dec 29 03:14:49.587430257 (563h45m49.458513302s) [upload=494284]: Next returned nil segment Dec 29 03:14:49.587447049 (563h45m49.458530084s) [upload=494284]: Waiting for error group managing segments... Dec 29 03:14:49.862457382 (563h45m49.733540417s) [noise=true , address=145.79.227.11:4729 , piece_id=V3JTWDCUENFAZ46KFCSJIWFMP6WL73JSNZU2CJEPNWKZMHRNPY5A, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.863652569 (563h45m49.734735604s) [noise=true , address=145.79.227.13:4802 , piece_id=XEND4N4V3WV6VQZ2CS3BMLYR4JTVFGNKBBO7DBCK3ROZW7J56ZYA, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.868734815 (563h45m49.73981785s) [noise=true , address=145.79.227.52:4445 , piece_id=X3QXKP5UAOM3VT6HHPG2MB6AZATTGFXJNYZFKV52IQ5HFOHOW3HA, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.882842599 (563h45m49.753925644s) [noise=true , address=145.79.227.2:4113 , piece_id=L4IIMJFDKZA2P77SL5EFAC4NQGOKPUBY3CP6AFJOMS5N4EFP4GTQ, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.883169571 (563h45m49.754252606s) [noise=true , address=145.79.227.15:4873 , piece_id=XM65EB6TF6YVZGWRA4DATYYLRYJMJZHM6C6XAJLFH27ZBZTOPX4A, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.885781053 (563h45m49.756864099s) [noise=true , address=145.79.227.47:4265 , piece_id=MVL5FDS6ATIWW5QZSKI27Y7LUHISG2ZHPBQ766U4T7Y2UEK6YLWA, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.895314616 (563h45m49.766397651s) [noise=true , address=145.79.227.48:4302 , piece_id=WHHBMYZGM4E6J53GZ4D4WX3F62GTN7BSGXBTJJLV3XOWMWZYLAJA, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.895798778 (563h45m49.766881823s) [noise=true , address=145.79.227.29:4941 , piece_id=XT3NRLFCWGODRZNNTFSRJQYJ2WGQSZRNVKF6YT3IKE2YMUULAC7A, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:49.903577400 (563h45m49.774660435s) [noise=true , address=145.79.227.36:4517 , piece_id=4TRR2Y2IWTXG4IHH3VE4Z2MGH6MF3CN4BUFHOYJPZMRAOFRQVA2A, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:50.052656715 (563h45m49.92373975s) [noise=true , address=77.90.31.10:4518 , piece_id=E2ITHUYLLMXKSIMXIFELWUQ6NKBPT2YA74H4KB4BCYJ7FHIKU6DQ, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:50.174339304 (563h45m50.045422339s) [noise=true , address=77.90.31.11:4555 , piece_id=5CLAKU6F2TES7YHTUG5ZM4FKYGDATEEXZZGAFJTEDL5KDBFTKKNA, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:50.198777748 (563h45m50.069860783s) [noise=true , address=77.90.31.19:4842 , piece_id=UKKYNA5BG3P24MHZSGMZWLOCHPHOEKNAI23LNZ7OJAUUNDLD62QQ, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:50.233522115 (563h45m50.10460515s) [noise=true , address=45.91.116.6:4976 , piece_id=L7ZF6WDJIO6RN7UVN6DIXK5OCANBUPEM3UZI4UWFABXGHS7T73TQ, seg_pos={0 0}, upload=494284]: Done uploading piece. err: <nil> Dec 29 03:14:50.233550439 (563h45m50.104633474s) [seg_pos={0 0}, upload=494284]: Segment reached optimal threshold of 13 pieces. Dec 29 03:14:50.233740181 (563h45m50.104823226s) [noise=true , address=145.79.227.19:4077 , piece_id=DIG35FEKP2N5H4UXGNJPY6S4NLLSKY3XLZIVZIQXMG7R4JXPDSDQ, seg_pos={0 0}, upload=494284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:47542->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:47542->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->145.79.227.19:4077: use of closed network connection Dec 29 03:14:50.233747164 (563h45m50.104830199s) [noise=true , address=145.79.227.19:4077 , piece_id=DIG35FEKP2N5H4UXGNJPY6S4NLLSKY3XLZIVZIQXMG7R4JXPDSDQ, seg_pos={0 0}, upload=494284]: Long tail cancellation due to optimal threshold Dec 29 03:14:50.234018611 (563h45m50.105101646s) [noise=true , address=45.91.116.12:4225 , piece_id=Y22G6GFWCXVOQM7RZCHTILKK52XXDJL2GEK2JJLIX5REWNQIGXUA, seg_pos={0 0}, upload=494284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 03:14:50.234025133 (563h45m50.105108168s) [noise=true , address=45.91.116.12:4225 , piece_id=Y22G6GFWCXVOQM7RZCHTILKK52XXDJL2GEK2JJLIX5REWNQIGXUA, seg_pos={0 0}, upload=494284]: Long tail cancellation due to optimal threshold Dec 29 03:14:50.234122349 (563h45m50.105205384s) [noise=true , address=45.91.116.7:4011 , piece_id=56B5LJUXG4JABE4MNI6T4LBKGCC6ATLCZTVBXRUC2OFLKFZWV5AQ, seg_pos={0 0}, upload=494284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 03:14:50.234127959 (563h45m50.105211004s) [noise=true , address=45.91.116.7:4011 , piece_id=56B5LJUXG4JABE4MNI6T4LBKGCC6ATLCZTVBXRUC2OFLKFZWV5AQ, seg_pos={0 0}, upload=494284]: Long tail cancellation due to optimal threshold Dec 29 03:14:50.234144420 (563h45m50.105227455s) [seg_pos={0 0}, upload=494284]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:14:50.234210436 (563h45m50.105293481s) [upload=494284]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:14:50.234214954 (563h45m50.105297989s) [upload=494284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:14:50.526543334 (563h45m50.397626389s) [upload=494284]: Done uploading segments... Dec 29 03:14:50.526564643 (563h45m50.397647679s) [upload=494284]: Upload finished. err: <nil> Duration: 30.004928975s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)