Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X.....X.X......................X...............] duration: 3.535079999s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:29:51 UTC) orbiter warning: context deadline exceeded Duration: 3.535272435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X......X.................X.............] duration: 28.382392785s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 28.382420458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.X.XX..X..XXX.X....X.X..X.....X...X....X..X..........] duration: 5.70888877s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:28:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.70891495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X......X.................X............] duration: 46.136630693s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:28:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 46.136654879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X.....X.X......................X..............] duration: 3.608663744s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:28:54 UTC) orbiter warning: context deadline exceeded Duration: 3.608693972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X.X.XX..X..XXX.X....X.X..X.....X...X....X..X.........] duration: 5.707355581s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.707374036s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............X.....X.X......................X.............] duration: 3.541173341s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:28:13 UTC) orbiter warning: context deadline exceeded Duration: 3.541244506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X......X.................X...........] duration: 51.722515503s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:28:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 51.722542895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X.....X.X......................X............] duration: 4.328584971s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:27:42 UTC) orbiter warning: context deadline exceeded Duration: 4.328608446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X........] duration: 5.76898324s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:27:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.76900971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................X.....X.X......................X...........] duration: 4.9995631s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:27:23 UTC) orbiter warning: context deadline exceeded Duration: 4.99958915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X......X.................X..........] duration: 52.427310094s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:26:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 52.427329892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X.......] duration: 5.694999422s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:26:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.695028277s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.................X.....X.X......................X..........] duration: 3.726214272s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:26:18 UTC) orbiter warning: context deadline exceeded Duration: 3.726242055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X......] duration: 5.690954531s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:26:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.690974107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X......X.................X.........] duration: 38.397602575s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:26:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 38.397617863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X.....] duration: 6.048677557s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:25:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 6.048707073s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.................X.....X.X......................X.........] duration: 3.551524082s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:25:34 UTC) orbiter warning: context deadline exceeded Duration: 3.551577894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X......X.................X........] duration: 39.955789612s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:25:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 39.955807435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X....] duration: 5.820196829s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:25:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.820227466s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.................X.....X.X......................X........] duration: 3.481210752s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:25:00 UTC) orbiter warning: context deadline exceeded Duration: 3.481235851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X......X.................X.......] duration: 47.511621616s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:24:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 47.511642205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X...] duration: 5.691163838s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:24:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.691191371s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X.................X.....X.X......................X.......] duration: 3.507699794s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:24:20 UTC) orbiter warning: context deadline exceeded Duration: 3.507727005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X..] duration: 6.014232128s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 6.014263206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X......X.................X......] duration: 58.555367144s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:23:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 58.555397973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X.................X.....X.X......................X......] duration: 3.586652551s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:23:50 UTC) orbiter warning: context deadline exceeded Duration: 3.586679835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X.................X.....X.X......................X.....] duration: 4.494126902s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:23:30 UTC) orbiter warning: context deadline exceeded Duration: 4.494153232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X.] duration: 5.73609256s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:23:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 5.736120334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..X] duration: 30.00305936s trace: https://jaeger.dev.storj.io/trace/6c0edfb9acfee1cc FAIL (2025-12-27 07:23:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ) context deadline exceeded ======LOGS====== Dec 27 07:22:41.784074322 (519h53m41.655157357s) [upload=456107]: Starting upload Dec 27 07:22:41.784138713 (519h53m41.655221748s) [upload=456107]: Done starting upload Dec 27 07:22:41.784292687 (519h53m41.655375722s) [upload=456107]: Uploading segments... Dec 27 07:22:41.784418346 (519h53m41.655501391s) [upload=456107]: Got next segment. Inline: false Dec 27 07:22:41.784459545 (519h53m41.65554258s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:22:41.784464935 (519h53m41.65554797s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:22:42.026406684 (519h53m41.897489739s) [seg_pos={0 0}, upload=456107]: Begin upload segment... Dec 27 07:22:42.026472980 (519h53m41.897556015s) [seg_pos={0 0}, upload=456107]: Done begin upload segment. Dec 27 07:22:42.026576177 (519h53m41.897659222s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.026606705 (519h53m41.89768975s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.027504304 (519h53m41.898587349s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028000158 (519h53m41.899083203s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028386143 (519h53m41.899469188s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.028795192 (519h53m41.899878237s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.029228647 (519h53m41.900311692s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.031941703 (519h53m41.903024738s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.032438029 (519h53m41.903521074s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033020407 (519h53m41.904103442s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033456178 (519h53m41.904539213s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.033888250 (519h53m41.904971295s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034324902 (519h53m41.905407947s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.034755481 (519h53m41.905838517s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.035144332 (519h53m41.906227377s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:42.039437758 (519h53m41.910521214s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Uploading piece... Dec 27 07:22:43.748373339 (519h53m43.619456374s) [upload=456107]: Next returned nil segment Dec 27 07:22:43.748385331 (519h53m43.619468367s) [upload=456107]: Waiting for error group managing segments... Dec 27 07:22:44.030870082 (519h53m43.901953117s) [noise=true , address=145.79.227.9:4366 , piece_id=L234ETPCEA5KHUPWWHM4CZIKCKNXPFMO4GA3TTXLP7JN4MGZZN7Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.040332751 (519h53m43.911415796s) [noise=true , address=145.79.227.4:4186 , piece_id=SEZ62TWVQHDZTB224F5E45RGKIHW2B4GD2JP5UB7MGS772W7DNIQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.041550139 (519h53m43.912633174s) [noise=true , address=145.79.227.51:4409 , piece_id=ODQXCFODNSL4CDCIUNKLKFTZ3JIN5UD4LO6MF6O7SJSVWGADMD2Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.043695273 (519h53m43.914778308s) [noise=true , address=145.79.227.29:4941 , piece_id=YMQEIJODSP5E4T65MPOFWRCD3QQEDR3H2WFPPBIXDBHUTFN74UBQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.047330055 (519h53m43.9184131s) [noise=true , address=145.79.227.41:4085 , piece_id=J633LAANFUE4XHKIBV6PP46FABBWWA5QPRQDA4YFDYPBSZWGZFZA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.050987129 (519h53m43.922070164s) [noise=true , address=145.79.227.49:4337 , piece_id=UTEXKII762CMAQVX4PKLWL27RMBNWKWPNVXASWOE5QMAI7HS4QLQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.057395111 (519h53m43.928478146s) [noise=true , address=145.79.227.5:4221 , piece_id=A6WQPMVOACHX3ZMR5XA54B74X24ZIDFMC4KTUVXRFS7DX4HD7T5Q, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.058219721 (519h53m43.929302756s) [noise=true , address=145.79.227.37:4567 , piece_id=LSNB35ONADQBCPKL5RVOIPASCOKXPFG2PF7B52PFN6USYPTYG5SQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.067076716 (519h53m43.938159771s) [noise=true , address=145.79.227.14:4840 , piece_id=VYF342BWLLVNQTSHNV5X37BJOL6MLTY6URF4O6EPFZOPZ4K2EBWQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.079140137 (519h53m43.950223172s) [noise=true , address=145.79.227.47:4265 , piece_id=WKKYNDTF5DJRV3RCJGBR4SEQNAU744VLAQRN5QQLXCD4BMNY4TMA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.115609190 (519h53m43.986692235s) [noise=true , address=145.79.227.13:4804 , piece_id=KH4PEYQPSL63ONYUCC2QZOATRD5A5ZEF4XT45YSQGFJTJQWGG4MA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.221138356 (519h53m44.092221391s) [noise=true , address=77.90.31.18:4808 , piece_id=KANKHI5IAMH3TZCVZ5DA2F5XXBA4V3WSOPRLBTSGOCRHROLK52DA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271408018 (519h53m44.142491053s) [noise=true , address=45.91.116.13:4156 , piece_id=V3EXZYQZC7OVPFYHBPLVBUWUDRE6RJSYTZUH4L7A4ZPVUX3NJJZQ, seg_pos={0 0}, upload=456107]: Done uploading piece. err: <nil> Dec 27 07:22:44.271433938 (519h53m44.142516983s) [seg_pos={0 0}, upload=456107]: Segment reached optimal threshold of 13 pieces. Dec 27 07:22:44.271727697 (519h53m44.142810732s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271737777 (519h53m44.142820812s) [noise=true , address=77.90.31.11:4556 , piece_id=Q5IBQFW2DQ6QYVTSQKD6RCKMTMJWEXSV7UXB22VKADDNCY33RFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.271901778 (519h53m44.142984813s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.271918491 (519h53m44.143001526s) [noise=true , address=45.91.116.7:4010 , piece_id=G2PGB2JTQD4MLKJICBZX5XOLVG6HB2EVM2DOWEM3NKBFIJ6ZWFUA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272086370 (519h53m44.143169405s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 07:22:44.272106789 (519h53m44.143189824s) [noise=true , address=77.90.31.10:4518 , piece_id=RRKU3KASC7PIQ6NSSZ7H4PA775D6QY4FKWJZ3WYLEGOEVIXLRXXA, seg_pos={0 0}, upload=456107]: Long tail cancellation due to optimal threshold Dec 27 07:22:44.272132738 (519h53m44.143215773s) [seg_pos={0 0}, upload=456107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:22:44.272233540 (519h53m44.143316586s) [upload=456107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:22:44.272241085 (519h53m44.14332412s) [upload=456107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:22:44.562930708 (519h53m44.434013743s) [upload=456107]: Done uploading segments... Dec 27 07:22:44.562950206 (519h53m44.434033251s) [upload=456107]: Upload finished. err: <nil> Duration: 30.003079108s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X......X.................X.....] duration: 38.294334763s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:22:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 38.294350644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X.....X.................X.....X.X......................X....] duration: 5.001336937s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:22:18 UTC) orbiter warning: context deadline exceeded Duration: 5.001359151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X......X.................X....] duration: 50.554235734s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:22:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 50.554251644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X..] duration: 5.803726657s trace: https://jaeger.dev.storj.io/trace/26c87245aa793211 FAIL (2025-12-27 07:21:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA) context deadline exceeded ======LOGS====== Dec 27 07:20:29.252172670 (519h51m29.123255715s) [upload=456075]: Starting upload Dec 27 07:20:29.252260015 (519h51m29.123343091s) [upload=456075]: Done starting upload Dec 27 07:20:29.252346220 (519h51m29.123429265s) [upload=456075]: Uploading segments... Dec 27 07:20:29.252560338 (519h51m29.123643383s) [upload=456075]: Got next segment. Inline: false Dec 27 07:20:29.252597178 (519h51m29.123680223s) [upload=456075]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:20:29.252602598 (519h51m29.123685633s) [upload=456075]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:20:29.499969418 (519h51m29.371052473s) [seg_pos={0 0}, upload=456075]: Begin upload segment... Dec 27 07:20:29.500022178 (519h51m29.371105223s) [seg_pos={0 0}, upload=456075]: Done begin upload segment. Dec 27 07:20:29.500053257 (519h51m29.371136302s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.500940336 (519h51m29.372023371s) [noise=true , address=145.79.227.41:4085 , piece_id=B2C2BEDCEPBLRGKV3FNKLKZNV556ZVBU4OXR7TEK74IYHF2OILKQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.501406494 (519h51m29.372489529s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.501892410 (519h51m29.372975455s) [noise=true , address=77.90.31.12:4590 , piece_id=WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.502331194 (519h51m29.37341424s) [noise=true , address=145.79.227.5:4221 , piece_id=3D45DQT3UYG7E5CMNT7MKR5UXGA24OXN2BEIIFS5FRGKMSUT3GDA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.502760553 (519h51m29.373843598s) [noise=true , address=145.79.227.12:4765 , piece_id=MNYMHRWH6F7RDTWEA3ZCDSLL553XXUVEPN42XS24QRJO6ZSNKR5Q, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.503208466 (519h51m29.374291511s) [noise=true , address=145.79.227.31:4798 , piece_id=5CUHNNJJFBEGHD2NKYFWQ64MXZE25JZPVFERVTKBWMT2VBPQY3CQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.504641414 (519h51m29.375724459s) [noise=true , address=145.79.227.37:4566 , piece_id=VFJTKGL7U3DDPOL4JJRQSFN7YAG3HJ323SBQFU4YI2VZMPO2YDYA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.506281988 (519h51m29.377365023s) [noise=true , address=145.79.227.36:4517 , piece_id=ZGBKKNUDO3ORH3JOHEVSUWU5YFJEZTOWCZF2TMSRGHUKKW7ATVWA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.508150917 (519h51m29.379233962s) [noise=true , address=45.91.116.7:4009 , piece_id=VF7JXTUNJXVTW76P7A6BH4BY5TMLSNQHYMC4C2EQNANHJ2CS7TXQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.509641545 (519h51m29.38072459s) [noise=true , address=145.79.227.11:4731 , piece_id=5LPPVM3KA7EXXETK3JB55P5ZI2RGYUNUI53Z3XBGGWZXPAJITE4A, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.511112647 (519h51m29.382195682s) [noise=true , address=45.91.116.14:4189 , piece_id=CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.511472962 (519h51m29.382556007s) [noise=true , address=145.79.227.29:4943 , piece_id=YUK7NF64KHW2XIUXILIZ4WITRLFOZTTKSEOW46FK6OGSU5ZU3E6A, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.513022804 (519h51m29.384105849s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.513351419 (519h51m29.384434454s) [noise=true , address=45.91.116.4:4940 , piece_id=3AXYECDEUZZQ4K2LMTZ65OBICIWXXT2OIRCZM2UGDQXKBXAMGNDQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.514893014 (519h51m29.38597606s) [noise=true , address=145.79.227.13:4804 , piece_id=AQGLPG4JB3GEM527QVPJ7R6FUAT2GFXCYJB4F6HWGKBHFE4I7OFA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:31.214328948 (519h51m31.085411993s) [upload=456075]: Next returned nil segment Dec 27 07:20:31.214345899 (519h51m31.085428944s) [upload=456075]: Waiting for error group managing segments... Dec 27 07:20:31.502450230 (519h51m31.373533265s) [noise=true , address=145.79.227.5:4221 , piece_id=3D45DQT3UYG7E5CMNT7MKR5UXGA24OXN2BEIIFS5FRGKMSUT3GDA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.518362801 (519h51m31.389445836s) [noise=true , address=145.79.227.11:4731 , piece_id=5LPPVM3KA7EXXETK3JB55P5ZI2RGYUNUI53Z3XBGGWZXPAJITE4A, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.518412235 (519h51m31.38949527s) [noise=true , address=145.79.227.36:4517 , piece_id=ZGBKKNUDO3ORH3JOHEVSUWU5YFJEZTOWCZF2TMSRGHUKKW7ATVWA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.531057414 (519h51m31.402140459s) [noise=true , address=145.79.227.37:4566 , piece_id=VFJTKGL7U3DDPOL4JJRQSFN7YAG3HJ323SBQFU4YI2VZMPO2YDYA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.532462709 (519h51m31.403545754s) [noise=true , address=145.79.227.12:4765 , piece_id=MNYMHRWH6F7RDTWEA3ZCDSLL553XXUVEPN42XS24QRJO6ZSNKR5Q, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.541155051 (519h51m31.412238086s) [noise=true , address=145.79.227.41:4085 , piece_id=B2C2BEDCEPBLRGKV3FNKLKZNV556ZVBU4OXR7TEK74IYHF2OILKQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.549497317 (519h51m31.420580362s) [noise=true , address=145.79.227.31:4798 , piece_id=5CUHNNJJFBEGHD2NKYFWQ64MXZE25JZPVFERVTKBWMT2VBPQY3CQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.554470988 (519h51m31.425554033s) [noise=true , address=145.79.227.29:4943 , piece_id=YUK7NF64KHW2XIUXILIZ4WITRLFOZTTKSEOW46FK6OGSU5ZU3E6A, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.695200201 (519h51m31.566283246s) [noise=true , address=77.90.31.12:4590 , piece_id=WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.711602353 (519h51m31.582685388s) [noise=true , address=145.79.227.13:4804 , piece_id=AQGLPG4JB3GEM527QVPJ7R6FUAT2GFXCYJB4F6HWGKBHFE4I7OFA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.766227358 (519h51m31.637310403s) [noise=true , address=45.91.116.4:4940 , piece_id=3AXYECDEUZZQ4K2LMTZ65OBICIWXXT2OIRCZM2UGDQXKBXAMGNDQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.786998831 (519h51m31.658081876s) [noise=true , address=45.91.116.7:4009 , piece_id=VF7JXTUNJXVTW76P7A6BH4BY5TMLSNQHYMC4C2EQNANHJ2CS7TXQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.789688843 (519h51m31.660771888s) [noise=true , address=45.91.116.14:4189 , piece_id=CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.789701897 (519h51m31.660784932s) [seg_pos={0 0}, upload=456075]: Segment reached optimal threshold of 13 pieces. Dec 27 07:20:31.789992171 (519h51m31.661075216s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790001248 (519h51m31.661084293s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790078945 (519h51m31.661161981s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790092722 (519h51m31.661175767s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790258637 (519h51m31.661341693s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790267756 (519h51m31.661350801s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790286731 (519h51m31.661369776s) [seg_pos={0 0}, upload=456075]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:20:31.790360101 (519h51m31.661443146s) [upload=456075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:20:31.790366394 (519h51m31.661449439s) [upload=456075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:20:32.079294252 (519h51m31.950377297s) [upload=456075]: Done uploading segments... Dec 27 07:20:32.079316846 (519h51m31.950399891s) [upload=456075]: Upload finished. err: <nil> Duration: 5.803749792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X.] duration: 5.97073964s trace: https://jaeger.dev.storj.io/trace/26c87245aa793211 FAIL (2025-12-27 07:21:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA) context deadline exceeded ======LOGS====== Dec 27 07:20:29.252172670 (519h51m29.123255715s) [upload=456075]: Starting upload Dec 27 07:20:29.252260015 (519h51m29.123343091s) [upload=456075]: Done starting upload Dec 27 07:20:29.252346220 (519h51m29.123429265s) [upload=456075]: Uploading segments... Dec 27 07:20:29.252560338 (519h51m29.123643383s) [upload=456075]: Got next segment. Inline: false Dec 27 07:20:29.252597178 (519h51m29.123680223s) [upload=456075]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:20:29.252602598 (519h51m29.123685633s) [upload=456075]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:20:29.499969418 (519h51m29.371052473s) [seg_pos={0 0}, upload=456075]: Begin upload segment... Dec 27 07:20:29.500022178 (519h51m29.371105223s) [seg_pos={0 0}, upload=456075]: Done begin upload segment. Dec 27 07:20:29.500053257 (519h51m29.371136302s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.500940336 (519h51m29.372023371s) [noise=true , address=145.79.227.41:4085 , piece_id=B2C2BEDCEPBLRGKV3FNKLKZNV556ZVBU4OXR7TEK74IYHF2OILKQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.501406494 (519h51m29.372489529s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.501892410 (519h51m29.372975455s) [noise=true , address=77.90.31.12:4590 , piece_id=WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.502331194 (519h51m29.37341424s) [noise=true , address=145.79.227.5:4221 , piece_id=3D45DQT3UYG7E5CMNT7MKR5UXGA24OXN2BEIIFS5FRGKMSUT3GDA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.502760553 (519h51m29.373843598s) [noise=true , address=145.79.227.12:4765 , piece_id=MNYMHRWH6F7RDTWEA3ZCDSLL553XXUVEPN42XS24QRJO6ZSNKR5Q, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.503208466 (519h51m29.374291511s) [noise=true , address=145.79.227.31:4798 , piece_id=5CUHNNJJFBEGHD2NKYFWQ64MXZE25JZPVFERVTKBWMT2VBPQY3CQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.504641414 (519h51m29.375724459s) [noise=true , address=145.79.227.37:4566 , piece_id=VFJTKGL7U3DDPOL4JJRQSFN7YAG3HJ323SBQFU4YI2VZMPO2YDYA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.506281988 (519h51m29.377365023s) [noise=true , address=145.79.227.36:4517 , piece_id=ZGBKKNUDO3ORH3JOHEVSUWU5YFJEZTOWCZF2TMSRGHUKKW7ATVWA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.508150917 (519h51m29.379233962s) [noise=true , address=45.91.116.7:4009 , piece_id=VF7JXTUNJXVTW76P7A6BH4BY5TMLSNQHYMC4C2EQNANHJ2CS7TXQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.509641545 (519h51m29.38072459s) [noise=true , address=145.79.227.11:4731 , piece_id=5LPPVM3KA7EXXETK3JB55P5ZI2RGYUNUI53Z3XBGGWZXPAJITE4A, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.511112647 (519h51m29.382195682s) [noise=true , address=45.91.116.14:4189 , piece_id=CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.511472962 (519h51m29.382556007s) [noise=true , address=145.79.227.29:4943 , piece_id=YUK7NF64KHW2XIUXILIZ4WITRLFOZTTKSEOW46FK6OGSU5ZU3E6A, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.513022804 (519h51m29.384105849s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.513351419 (519h51m29.384434454s) [noise=true , address=45.91.116.4:4940 , piece_id=3AXYECDEUZZQ4K2LMTZ65OBICIWXXT2OIRCZM2UGDQXKBXAMGNDQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.514893014 (519h51m29.38597606s) [noise=true , address=145.79.227.13:4804 , piece_id=AQGLPG4JB3GEM527QVPJ7R6FUAT2GFXCYJB4F6HWGKBHFE4I7OFA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:31.214328948 (519h51m31.085411993s) [upload=456075]: Next returned nil segment Dec 27 07:20:31.214345899 (519h51m31.085428944s) [upload=456075]: Waiting for error group managing segments... Dec 27 07:20:31.502450230 (519h51m31.373533265s) [noise=true , address=145.79.227.5:4221 , piece_id=3D45DQT3UYG7E5CMNT7MKR5UXGA24OXN2BEIIFS5FRGKMSUT3GDA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.518362801 (519h51m31.389445836s) [noise=true , address=145.79.227.11:4731 , piece_id=5LPPVM3KA7EXXETK3JB55P5ZI2RGYUNUI53Z3XBGGWZXPAJITE4A, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.518412235 (519h51m31.38949527s) [noise=true , address=145.79.227.36:4517 , piece_id=ZGBKKNUDO3ORH3JOHEVSUWU5YFJEZTOWCZF2TMSRGHUKKW7ATVWA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.531057414 (519h51m31.402140459s) [noise=true , address=145.79.227.37:4566 , piece_id=VFJTKGL7U3DDPOL4JJRQSFN7YAG3HJ323SBQFU4YI2VZMPO2YDYA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.532462709 (519h51m31.403545754s) [noise=true , address=145.79.227.12:4765 , piece_id=MNYMHRWH6F7RDTWEA3ZCDSLL553XXUVEPN42XS24QRJO6ZSNKR5Q, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.541155051 (519h51m31.412238086s) [noise=true , address=145.79.227.41:4085 , piece_id=B2C2BEDCEPBLRGKV3FNKLKZNV556ZVBU4OXR7TEK74IYHF2OILKQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.549497317 (519h51m31.420580362s) [noise=true , address=145.79.227.31:4798 , piece_id=5CUHNNJJFBEGHD2NKYFWQ64MXZE25JZPVFERVTKBWMT2VBPQY3CQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.554470988 (519h51m31.425554033s) [noise=true , address=145.79.227.29:4943 , piece_id=YUK7NF64KHW2XIUXILIZ4WITRLFOZTTKSEOW46FK6OGSU5ZU3E6A, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.695200201 (519h51m31.566283246s) [noise=true , address=77.90.31.12:4590 , piece_id=WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.711602353 (519h51m31.582685388s) [noise=true , address=145.79.227.13:4804 , piece_id=AQGLPG4JB3GEM527QVPJ7R6FUAT2GFXCYJB4F6HWGKBHFE4I7OFA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.766227358 (519h51m31.637310403s) [noise=true , address=45.91.116.4:4940 , piece_id=3AXYECDEUZZQ4K2LMTZ65OBICIWXXT2OIRCZM2UGDQXKBXAMGNDQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.786998831 (519h51m31.658081876s) [noise=true , address=45.91.116.7:4009 , piece_id=VF7JXTUNJXVTW76P7A6BH4BY5TMLSNQHYMC4C2EQNANHJ2CS7TXQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.789688843 (519h51m31.660771888s) [noise=true , address=45.91.116.14:4189 , piece_id=CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.789701897 (519h51m31.660784932s) [seg_pos={0 0}, upload=456075]: Segment reached optimal threshold of 13 pieces. Dec 27 07:20:31.789992171 (519h51m31.661075216s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790001248 (519h51m31.661084293s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790078945 (519h51m31.661161981s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790092722 (519h51m31.661175767s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790258637 (519h51m31.661341693s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790267756 (519h51m31.661350801s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790286731 (519h51m31.661369776s) [seg_pos={0 0}, upload=456075]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:20:31.790360101 (519h51m31.661443146s) [upload=456075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:20:31.790366394 (519h51m31.661449439s) [upload=456075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:20:32.079294252 (519h51m31.950377297s) [upload=456075]: Done uploading segments... Dec 27 07:20:32.079316846 (519h51m31.950399891s) [upload=456075]: Upload finished. err: <nil> Duration: 5.970766361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X.....X.................X.....X.X......................X...] duration: 3.724986274s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:21:11 UTC) orbiter warning: context deadline exceeded Duration: 3.725016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X......X.................X...] duration: 40.450550395s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 40.450575843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XXX...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....X] duration: 30.003561927s trace: https://jaeger.dev.storj.io/trace/26c87245aa793211 FAIL (2025-12-27 07:20:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA) context deadline exceeded ======LOGS====== Dec 27 07:20:29.252172670 (519h51m29.123255715s) [upload=456075]: Starting upload Dec 27 07:20:29.252260015 (519h51m29.123343091s) [upload=456075]: Done starting upload Dec 27 07:20:29.252346220 (519h51m29.123429265s) [upload=456075]: Uploading segments... Dec 27 07:20:29.252560338 (519h51m29.123643383s) [upload=456075]: Got next segment. Inline: false Dec 27 07:20:29.252597178 (519h51m29.123680223s) [upload=456075]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:20:29.252602598 (519h51m29.123685633s) [upload=456075]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:20:29.499969418 (519h51m29.371052473s) [seg_pos={0 0}, upload=456075]: Begin upload segment... Dec 27 07:20:29.500022178 (519h51m29.371105223s) [seg_pos={0 0}, upload=456075]: Done begin upload segment. Dec 27 07:20:29.500053257 (519h51m29.371136302s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.500940336 (519h51m29.372023371s) [noise=true , address=145.79.227.41:4085 , piece_id=B2C2BEDCEPBLRGKV3FNKLKZNV556ZVBU4OXR7TEK74IYHF2OILKQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.501406494 (519h51m29.372489529s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.501892410 (519h51m29.372975455s) [noise=true , address=77.90.31.12:4590 , piece_id=WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.502331194 (519h51m29.37341424s) [noise=true , address=145.79.227.5:4221 , piece_id=3D45DQT3UYG7E5CMNT7MKR5UXGA24OXN2BEIIFS5FRGKMSUT3GDA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.502760553 (519h51m29.373843598s) [noise=true , address=145.79.227.12:4765 , piece_id=MNYMHRWH6F7RDTWEA3ZCDSLL553XXUVEPN42XS24QRJO6ZSNKR5Q, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.503208466 (519h51m29.374291511s) [noise=true , address=145.79.227.31:4798 , piece_id=5CUHNNJJFBEGHD2NKYFWQ64MXZE25JZPVFERVTKBWMT2VBPQY3CQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.504641414 (519h51m29.375724459s) [noise=true , address=145.79.227.37:4566 , piece_id=VFJTKGL7U3DDPOL4JJRQSFN7YAG3HJ323SBQFU4YI2VZMPO2YDYA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.506281988 (519h51m29.377365023s) [noise=true , address=145.79.227.36:4517 , piece_id=ZGBKKNUDO3ORH3JOHEVSUWU5YFJEZTOWCZF2TMSRGHUKKW7ATVWA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.508150917 (519h51m29.379233962s) [noise=true , address=45.91.116.7:4009 , piece_id=VF7JXTUNJXVTW76P7A6BH4BY5TMLSNQHYMC4C2EQNANHJ2CS7TXQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.509641545 (519h51m29.38072459s) [noise=true , address=145.79.227.11:4731 , piece_id=5LPPVM3KA7EXXETK3JB55P5ZI2RGYUNUI53Z3XBGGWZXPAJITE4A, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.511112647 (519h51m29.382195682s) [noise=true , address=45.91.116.14:4189 , piece_id=CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.511472962 (519h51m29.382556007s) [noise=true , address=145.79.227.29:4943 , piece_id=YUK7NF64KHW2XIUXILIZ4WITRLFOZTTKSEOW46FK6OGSU5ZU3E6A, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.513022804 (519h51m29.384105849s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.513351419 (519h51m29.384434454s) [noise=true , address=45.91.116.4:4940 , piece_id=3AXYECDEUZZQ4K2LMTZ65OBICIWXXT2OIRCZM2UGDQXKBXAMGNDQ, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:29.514893014 (519h51m29.38597606s) [noise=true , address=145.79.227.13:4804 , piece_id=AQGLPG4JB3GEM527QVPJ7R6FUAT2GFXCYJB4F6HWGKBHFE4I7OFA, seg_pos={0 0}, upload=456075]: Uploading piece... Dec 27 07:20:31.214328948 (519h51m31.085411993s) [upload=456075]: Next returned nil segment Dec 27 07:20:31.214345899 (519h51m31.085428944s) [upload=456075]: Waiting for error group managing segments... Dec 27 07:20:31.502450230 (519h51m31.373533265s) [noise=true , address=145.79.227.5:4221 , piece_id=3D45DQT3UYG7E5CMNT7MKR5UXGA24OXN2BEIIFS5FRGKMSUT3GDA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.518362801 (519h51m31.389445836s) [noise=true , address=145.79.227.11:4731 , piece_id=5LPPVM3KA7EXXETK3JB55P5ZI2RGYUNUI53Z3XBGGWZXPAJITE4A, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.518412235 (519h51m31.38949527s) [noise=true , address=145.79.227.36:4517 , piece_id=ZGBKKNUDO3ORH3JOHEVSUWU5YFJEZTOWCZF2TMSRGHUKKW7ATVWA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.531057414 (519h51m31.402140459s) [noise=true , address=145.79.227.37:4566 , piece_id=VFJTKGL7U3DDPOL4JJRQSFN7YAG3HJ323SBQFU4YI2VZMPO2YDYA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.532462709 (519h51m31.403545754s) [noise=true , address=145.79.227.12:4765 , piece_id=MNYMHRWH6F7RDTWEA3ZCDSLL553XXUVEPN42XS24QRJO6ZSNKR5Q, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.541155051 (519h51m31.412238086s) [noise=true , address=145.79.227.41:4085 , piece_id=B2C2BEDCEPBLRGKV3FNKLKZNV556ZVBU4OXR7TEK74IYHF2OILKQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.549497317 (519h51m31.420580362s) [noise=true , address=145.79.227.31:4798 , piece_id=5CUHNNJJFBEGHD2NKYFWQ64MXZE25JZPVFERVTKBWMT2VBPQY3CQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.554470988 (519h51m31.425554033s) [noise=true , address=145.79.227.29:4943 , piece_id=YUK7NF64KHW2XIUXILIZ4WITRLFOZTTKSEOW46FK6OGSU5ZU3E6A, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.695200201 (519h51m31.566283246s) [noise=true , address=77.90.31.12:4590 , piece_id=WBIOY6ROU4VO4BJ6PP77UPF2UNKVLHMOJMXDXHPK5AIWMNON4ZJA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.711602353 (519h51m31.582685388s) [noise=true , address=145.79.227.13:4804 , piece_id=AQGLPG4JB3GEM527QVPJ7R6FUAT2GFXCYJB4F6HWGKBHFE4I7OFA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.766227358 (519h51m31.637310403s) [noise=true , address=45.91.116.4:4940 , piece_id=3AXYECDEUZZQ4K2LMTZ65OBICIWXXT2OIRCZM2UGDQXKBXAMGNDQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.786998831 (519h51m31.658081876s) [noise=true , address=45.91.116.7:4009 , piece_id=VF7JXTUNJXVTW76P7A6BH4BY5TMLSNQHYMC4C2EQNANHJ2CS7TXQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.789688843 (519h51m31.660771888s) [noise=true , address=45.91.116.14:4189 , piece_id=CB4OIHLPK4SNJZAMLK4LWXB22HZQD77GPZPEXAXZJJ4IAX6IHCFQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: <nil> Dec 27 07:20:31.789701897 (519h51m31.660784932s) [seg_pos={0 0}, upload=456075]: Segment reached optimal threshold of 13 pieces. Dec 27 07:20:31.789992171 (519h51m31.661075216s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790001248 (519h51m31.661084293s) [noise=true , address=45.91.116.13:4156 , piece_id=XTBIEF67NFFGQYLVT6G6MAF554E4VM5TW4CP2VBQWR54ELRQO4RA, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790078945 (519h51m31.661161981s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790092722 (519h51m31.661175767s) [noise=true , address=45.91.116.10:4083 , piece_id=J3SQRZIUINDMBGJMCHDESGLO4G6U6GNHDIWSZ6FNZLPBZTIMP6DQ, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790258637 (519h51m31.661341693s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 07:20:31.790267756 (519h51m31.661350801s) [noise=true , address=45.91.116.5:4298 , piece_id=IJPK7VIPIRBGMZFCGRKKMUU2KCJ27I76QFCZOUUQOEGT5VY2HTQA, seg_pos={0 0}, upload=456075]: Long tail cancellation due to optimal threshold Dec 27 07:20:31.790286731 (519h51m31.661369776s) [seg_pos={0 0}, upload=456075]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:20:31.790360101 (519h51m31.661443146s) [upload=456075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:20:31.790366394 (519h51m31.661449439s) [upload=456075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:20:32.079294252 (519h51m31.950377297s) [upload=456075]: Done uploading segments... Dec 27 07:20:32.079316846 (519h51m31.950399891s) [upload=456075]: Upload finished. err: <nil> Duration: 30.003588727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X.....X.................X.....X.X......................X..] duration: 3.521928978s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:20:50 UTC) orbiter warning: context deadline exceeded Duration: 3.521953975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X.....X.................X.....X.X......................X.] duration: 4.916655591s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:20:33 UTC) orbiter warning: context deadline exceeded Duration: 4.916681629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X......X.................X..] duration: 40.568284382s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 40.568300361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXX...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X....] duration: 5.963664177s trace: https://jaeger.dev.storj.io/trace/354bcb7f92000dfd FAIL (2025-12-27 07:20:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA) context deadline exceeded ======LOGS====== :36.484181944 (519h48m36.355264989s) [upload=456027]: Uploading segments... Dec 27 07:17:36.484440587 (519h48m36.355523632s) [upload=456027]: Got next segment. Inline: false Dec 27 07:17:36.484490993 (519h48m36.355574028s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:17:36.484496954 (519h48m36.355579999s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:17:36.734283197 (519h48m36.605366242s) [seg_pos={0 0}, upload=456027]: Begin upload segment... Dec 27 07:17:36.734328443 (519h48m36.605411478s) [seg_pos={0 0}, upload=456027]: Done begin upload segment. Dec 27 07:17:36.734361326 (519h48m36.605444361s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.734472347 (519h48m36.605555402s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735137814 (519h48m36.606220849s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735589384 (519h48m36.606672429s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.736795782 (519h48m36.607878827s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.737479052 (519h48m36.608562097s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738242376 (519h48m36.609325431s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738295818 (519h48m36.609378853s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739051005 (519h48m36.61013404s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739074571 (519h48m36.610157616s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739845478 (519h48m36.610928523s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.742246831 (519h48m36.613329866s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.744437011 (519h48m36.615520047s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.746960607 (519h48m36.618043652s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.750350071 (519h48m36.621433106s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.754205573 (519h48m36.625288618s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:38.350472332 (519h48m38.221555367s) [upload=456027]: Next returned nil segment Dec 27 07:17:38.350483994 (519h48m38.221567029s) [upload=456027]: Waiting for error group managing segments... Dec 27 07:17:38.736633910 (519h48m38.607716975s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742033872 (519h48m38.613116917s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742420859 (519h48m38.613503904s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.751825047 (519h48m38.622908092s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.758444301 (519h48m38.629527336s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.761533484 (519h48m38.632616519s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.767038246 (519h48m38.638121291s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.769882512 (519h48m38.640965557s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.779094093 (519h48m38.650177138s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.782769932 (519h48m38.653852978s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.790884684 (519h48m38.661967729s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.082544603 (519h48m38.953627648s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124212186 (519h48m38.995295221s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124233447 (519h48m38.995316482s) [seg_pos={0 0}, upload=456027]: Segment reached optimal threshold of 13 pieces. Dec 27 07:17:39.124442325 (519h48m38.99552536s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection Dec 27 07:17:39.124448436 (519h48m38.995531481s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124565418 (519h48m38.995648453s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:39.124582241 (519h48m38.995665276s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124808872 (519h48m38.995891907s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: context canceled Dec 27 07:17:39.124815935 (519h48m38.99589897s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124834341 (519h48m38.995917376s) [seg_pos={0 0}, upload=456027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:17:39.124902180 (519h48m38.995985215s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:39.124906648 (519h48m38.995989693s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:39.425753951 (519h48m39.296836986s) [upload=456027]: Done uploading segments... Dec 27 07:17:39.425778739 (519h48m39.296861784s) [upload=456027]: Upload finished. err: <nil> Duration: 5.963716316s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X.....X.................X.....X.X......................X] duration: 5.001034281s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:19:50 UTC) orbiter warning: context deadline exceeded Duration: 5.001053688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X......X.................X.] duration: 52.538074969s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:19:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 52.538091261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XXX...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X...] duration: 5.657709445s trace: https://jaeger.dev.storj.io/trace/354bcb7f92000dfd FAIL (2025-12-27 07:19:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA) context deadline exceeded ======LOGS====== :36.484181944 (519h48m36.355264989s) [upload=456027]: Uploading segments... Dec 27 07:17:36.484440587 (519h48m36.355523632s) [upload=456027]: Got next segment. Inline: false Dec 27 07:17:36.484490993 (519h48m36.355574028s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:17:36.484496954 (519h48m36.355579999s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:17:36.734283197 (519h48m36.605366242s) [seg_pos={0 0}, upload=456027]: Begin upload segment... Dec 27 07:17:36.734328443 (519h48m36.605411478s) [seg_pos={0 0}, upload=456027]: Done begin upload segment. Dec 27 07:17:36.734361326 (519h48m36.605444361s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.734472347 (519h48m36.605555402s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735137814 (519h48m36.606220849s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735589384 (519h48m36.606672429s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.736795782 (519h48m36.607878827s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.737479052 (519h48m36.608562097s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738242376 (519h48m36.609325431s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738295818 (519h48m36.609378853s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739051005 (519h48m36.61013404s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739074571 (519h48m36.610157616s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739845478 (519h48m36.610928523s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.742246831 (519h48m36.613329866s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.744437011 (519h48m36.615520047s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.746960607 (519h48m36.618043652s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.750350071 (519h48m36.621433106s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.754205573 (519h48m36.625288618s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:38.350472332 (519h48m38.221555367s) [upload=456027]: Next returned nil segment Dec 27 07:17:38.350483994 (519h48m38.221567029s) [upload=456027]: Waiting for error group managing segments... Dec 27 07:17:38.736633910 (519h48m38.607716975s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742033872 (519h48m38.613116917s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742420859 (519h48m38.613503904s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.751825047 (519h48m38.622908092s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.758444301 (519h48m38.629527336s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.761533484 (519h48m38.632616519s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.767038246 (519h48m38.638121291s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.769882512 (519h48m38.640965557s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.779094093 (519h48m38.650177138s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.782769932 (519h48m38.653852978s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.790884684 (519h48m38.661967729s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.082544603 (519h48m38.953627648s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124212186 (519h48m38.995295221s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124233447 (519h48m38.995316482s) [seg_pos={0 0}, upload=456027]: Segment reached optimal threshold of 13 pieces. Dec 27 07:17:39.124442325 (519h48m38.99552536s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection Dec 27 07:17:39.124448436 (519h48m38.995531481s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124565418 (519h48m38.995648453s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:39.124582241 (519h48m38.995665276s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124808872 (519h48m38.995891907s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: context canceled Dec 27 07:17:39.124815935 (519h48m38.99589897s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124834341 (519h48m38.995917376s) [seg_pos={0 0}, upload=456027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:17:39.124902180 (519h48m38.995985215s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:39.124906648 (519h48m38.995989693s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:39.425753951 (519h48m39.296836986s) [upload=456027]: Done uploading segments... Dec 27 07:17:39.425778739 (519h48m39.296861784s) [upload=456027]: Upload finished. err: <nil> Duration: 5.657741858s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X.....X.................X.....X.X......................] duration: 3.494770502s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:19:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494788938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X..] duration: 5.733942176s trace: https://jaeger.dev.storj.io/trace/354bcb7f92000dfd FAIL (2025-12-27 07:19:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA) context deadline exceeded ======LOGS====== :36.484181944 (519h48m36.355264989s) [upload=456027]: Uploading segments... Dec 27 07:17:36.484440587 (519h48m36.355523632s) [upload=456027]: Got next segment. Inline: false Dec 27 07:17:36.484490993 (519h48m36.355574028s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:17:36.484496954 (519h48m36.355579999s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:17:36.734283197 (519h48m36.605366242s) [seg_pos={0 0}, upload=456027]: Begin upload segment... Dec 27 07:17:36.734328443 (519h48m36.605411478s) [seg_pos={0 0}, upload=456027]: Done begin upload segment. Dec 27 07:17:36.734361326 (519h48m36.605444361s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.734472347 (519h48m36.605555402s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735137814 (519h48m36.606220849s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735589384 (519h48m36.606672429s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.736795782 (519h48m36.607878827s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.737479052 (519h48m36.608562097s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738242376 (519h48m36.609325431s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738295818 (519h48m36.609378853s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739051005 (519h48m36.61013404s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739074571 (519h48m36.610157616s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739845478 (519h48m36.610928523s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.742246831 (519h48m36.613329866s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.744437011 (519h48m36.615520047s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.746960607 (519h48m36.618043652s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.750350071 (519h48m36.621433106s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.754205573 (519h48m36.625288618s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:38.350472332 (519h48m38.221555367s) [upload=456027]: Next returned nil segment Dec 27 07:17:38.350483994 (519h48m38.221567029s) [upload=456027]: Waiting for error group managing segments... Dec 27 07:17:38.736633910 (519h48m38.607716975s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742033872 (519h48m38.613116917s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742420859 (519h48m38.613503904s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.751825047 (519h48m38.622908092s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.758444301 (519h48m38.629527336s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.761533484 (519h48m38.632616519s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.767038246 (519h48m38.638121291s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.769882512 (519h48m38.640965557s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.779094093 (519h48m38.650177138s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.782769932 (519h48m38.653852978s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.790884684 (519h48m38.661967729s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.082544603 (519h48m38.953627648s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124212186 (519h48m38.995295221s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124233447 (519h48m38.995316482s) [seg_pos={0 0}, upload=456027]: Segment reached optimal threshold of 13 pieces. Dec 27 07:17:39.124442325 (519h48m38.99552536s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection Dec 27 07:17:39.124448436 (519h48m38.995531481s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124565418 (519h48m38.995648453s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:39.124582241 (519h48m38.995665276s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124808872 (519h48m38.995891907s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: context canceled Dec 27 07:17:39.124815935 (519h48m38.99589897s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124834341 (519h48m38.995917376s) [seg_pos={0 0}, upload=456027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:17:39.124902180 (519h48m38.995985215s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:39.124906648 (519h48m38.995989693s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:39.425753951 (519h48m39.296836986s) [upload=456027]: Done uploading segments... Dec 27 07:17:39.425778739 (519h48m39.296861784s) [upload=456027]: Upload finished. err: <nil> Duration: 5.733966683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X......X.................X] duration: 1m0.003466498s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 1m0.003477879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......X.....X.................X.....X.X.....................] duration: 3.603406675s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:18:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.603431922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X..X.....X...X.] duration: 6.036529355s trace: https://jaeger.dev.storj.io/trace/354bcb7f92000dfd FAIL (2025-12-27 07:18:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA) context deadline exceeded ======LOGS====== :36.484181944 (519h48m36.355264989s) [upload=456027]: Uploading segments... Dec 27 07:17:36.484440587 (519h48m36.355523632s) [upload=456027]: Got next segment. Inline: false Dec 27 07:17:36.484490993 (519h48m36.355574028s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:17:36.484496954 (519h48m36.355579999s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:17:36.734283197 (519h48m36.605366242s) [seg_pos={0 0}, upload=456027]: Begin upload segment... Dec 27 07:17:36.734328443 (519h48m36.605411478s) [seg_pos={0 0}, upload=456027]: Done begin upload segment. Dec 27 07:17:36.734361326 (519h48m36.605444361s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.734472347 (519h48m36.605555402s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735137814 (519h48m36.606220849s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.735589384 (519h48m36.606672429s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.736795782 (519h48m36.607878827s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.737479052 (519h48m36.608562097s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738242376 (519h48m36.609325431s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.738295818 (519h48m36.609378853s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739051005 (519h48m36.61013404s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739074571 (519h48m36.610157616s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.739845478 (519h48m36.610928523s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.742246831 (519h48m36.613329866s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.744437011 (519h48m36.615520047s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.746960607 (519h48m36.618043652s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.750350071 (519h48m36.621433106s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:36.754205573 (519h48m36.625288618s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Uploading piece... Dec 27 07:17:38.350472332 (519h48m38.221555367s) [upload=456027]: Next returned nil segment Dec 27 07:17:38.350483994 (519h48m38.221567029s) [upload=456027]: Waiting for error group managing segments... Dec 27 07:17:38.736633910 (519h48m38.607716975s) [noise=true , address=145.79.227.15:4873 , piece_id=TSKJYJU4SEMYT2NXUQHU6VWPCNUJZ7OXB5MK33Q3BGRF4Z7N5RMQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742033872 (519h48m38.613116917s) [noise=true , address=145.79.227.31:4797 , piece_id=2IBPCFMLDHOQYNRDJE3CJGNZGY7YCRQ6W4F6FQC34TILASXDPK5A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.742420859 (519h48m38.613503904s) [noise=true , address=145.79.227.6:4259 , piece_id=54WZCLP7HWVLVBNC466WIRWXEMAHDRJME4S7QOXGCB3236KDKLUQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.751825047 (519h48m38.622908092s) [noise=true , address=145.79.227.39:4015 , piece_id=NXXYV7KKVQC3XT56LFGWQDQCU6DRF34XEXSOBHWHVDWCPEVPXAZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.758444301 (519h48m38.629527336s) [noise=true , address=145.79.227.50:4375 , piece_id=F6EKHX5OBWTAQFV7KJSVXPQAOXD2H6AQNHVK57NGTOLMUS3RAJLA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.761533484 (519h48m38.632616519s) [noise=true , address=145.79.227.36:4517 , piece_id=WG2VMRIWLIYGUBWPVCDMYZW4JKHLPGXFJ2U66XI2T5KELP75322Q, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.767038246 (519h48m38.638121291s) [noise=true , address=145.79.227.2:4113 , piece_id=BVWUC2O5YFT56HQXDKLCKLOT3M7AI7X4H7I4TYAZNDZU6JI3FG2A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.769882512 (519h48m38.640965557s) [noise=true , address=145.79.227.25:4617 , piece_id=WHFA25SBBWQPYJXCGMJWAKSV7JV7DAPVGDXA3TNJLOZSVHGKF2YA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.779094093 (519h48m38.650177138s) [noise=true , address=145.79.227.8:4331 , piece_id=46K5AEKQLEBLRQERGXXM5DDDLKO4SIV7O4T3AXRJU3MBMMIXPA6A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.782769932 (519h48m38.653852978s) [noise=true , address=145.79.227.35:4482 , piece_id=42BJAAWIYM2ITBNMZHB3FHEY2VHK4ERTHIACSR3J5YQUONT4ZTWQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:38.790884684 (519h48m38.661967729s) [noise=true , address=145.79.227.14:4840 , piece_id=X2XB5ZNQAVIT6RUMVYPTUEOKEABTGQOOYKW2G2NHS3ATLNA7R6TA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.082544603 (519h48m38.953627648s) [noise=true , address=77.90.31.15:4697 , piece_id=AS5FLKZGJXTR4J7DRXLB3RQDQH5YLD3ALQGK4T7NWZWWYFOC7MGA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124212186 (519h48m38.995295221s) [noise=true , address=45.91.116.8:4046 , piece_id=IUMJZVOF5MAY2AF7PJDZMUWQT5ZGTAHVFALMCJVIHZTGWFZCMFAA, seg_pos={0 0}, upload=456027]: Done uploading piece. err: <nil> Dec 27 07:17:39.124233447 (519h48m38.995316482s) [seg_pos={0 0}, upload=456027]: Segment reached optimal threshold of 13 pieces. Dec 27 07:17:39.124442325 (519h48m38.99552536s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44808->77.90.31.19:4844: use of closed network connection Dec 27 07:17:39.124448436 (519h48m38.995531481s) [noise=true , address=77.90.31.19:4844 , piece_id=H243TOBMZGLMJEH2EK6BLWS74YTCJWKXPXSSDYTIUIUDS3H7SGNQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124565418 (519h48m38.995648453s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:39.124582241 (519h48m38.995665276s) [noise=true , address=77.90.31.21:4877 , piece_id=CZVEXR3BY3KSG6CHQT7GOQWBYHQQRZNK5RHMDRP5PAZDHVDQXO4A, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124808872 (519h48m38.995891907s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:35736->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: context canceled Dec 27 07:17:39.124815935 (519h48m38.99589897s) [noise=true , address=45.91.116.12:4225 , piece_id=SUAHI6WE5BGPU763FG77BBI5SSOQISBAB65SBWRH7N4AMPXYWTZQ, seg_pos={0 0}, upload=456027]: Long tail cancellation due to optimal threshold Dec 27 07:17:39.124834341 (519h48m38.995917376s) [seg_pos={0 0}, upload=456027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:17:39.124902180 (519h48m38.995985215s) [upload=456027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:39.124906648 (519h48m38.995989693s) [upload=456027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:39.425753951 (519h48m39.296836986s) [upload=456027]: Done uploading segments... Dec 27 07:17:39.425778739 (519h48m39.296861784s) [upload=456027]: Upload finished. err: <nil> Duration: 6.036549492s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)