Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.XXXX..X.] duration: 4.300951119s trace: https://jaeger.dev.storj.io/trace/241ad6da60babde8 FAIL (2025-12-29 17:14:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.300975727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...XXXX.XX.] duration: 6.027762s trace: https://jaeger.dev.storj.io/trace/2a0c7ab311525271 FAIL (2025-12-29 17:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 3SKFKEKPXPJH3W4W3TZXMROPPES2I4B2WXFXAWBQM5NHTCDXM6PA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: NAAAMF7T3IL44CM3OE56AGEPJWLAPTBCQYR4YT4F3DPJUAFX4RWQ) context deadline exceeded ======LOGS====== 5.583641395s) [upload=506457]: Uploading segments... Dec 29 17:13:05.712739534 (577h44m5.583822579s) [upload=506457]: Got next segment. Inline: false Dec 29 17:13:05.712768079 (577h44m5.583851124s) [upload=506457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:13:05.712772197 (577h44m5.583855232s) [upload=506457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:13:05.969718230 (577h44m5.840801305s) [seg_pos={0 0}, upload=506457]: Begin upload segment... Dec 29 17:13:05.969790878 (577h44m5.840873923s) [seg_pos={0 0}, upload=506457]: Done begin upload segment. Dec 29 17:13:05.969842436 (577h44m5.840925471s) [noise=true , address=45.91.116.11:4117 , piece_id=3SKFKEKPXPJH3W4W3TZXMROPPES2I4B2WXFXAWBQM5NHTCDXM6PA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.969873776 (577h44m5.840956821s) [noise=true , address=45.91.116.14:4190 , piece_id=O7KQBNCOJGP5J3FQIJEMJ7LR3PWNFZCDJHM4GCMWXGGO2PRQZ5DQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.969948157 (577h44m5.841031202s) [noise=true , address=145.79.227.38:4977 , piece_id=5W3CPOBK3BQ6EEWNTO47BHRFEJQSHM7OGM234FUPJ5MYL5R2JKZA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.970407653 (577h44m5.841490688s) [noise=true , address=77.90.31.13:4626 , piece_id=JH2MNAHTSRVOTJLZYH3ZJRSCGAYANO6SK5EODZ6GOVGXY3ETAKHA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.970749062 (577h44m5.841832107s) [noise=true , address=145.79.227.45:4229 , piece_id=IBBXISAUSWPD6PDBWXYBYRUNZURFLIUTFJRAQJNEXSMKEQEVPR5Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.971158402 (577h44m5.842241447s) [noise=true , address=145.79.227.12:4765 , piece_id=NLNBMVDR6KIS45K2CGRPYQOCJOXA2B4R6R6WK754ASGVEL52S75Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.971514339 (577h44m5.842597385s) [noise=true , address=145.79.227.15:4874 , piece_id=66YNLCBU5Y3KN3X3DDIKWSDDKZY7556DG3U2IVMCXA6VONFNBQSA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.972152135 (577h44m5.84323518s) [noise=true , address=145.79.227.14:4837 , piece_id=NKPQFIZQYTJ72K2B7UGAIKM5N6JY6YQ7R22KMERFGUOUTBG5STEA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.972796942 (577h44m5.843879987s) [noise=true , address=145.79.227.8:4329 , piece_id=LELWF5PB43OACEXDW3HGD7L6CWP3SV7VRHSAYNTGA35DZFKZZDZA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.973446499 (577h44m5.844529544s) [noise=true , address=77.90.31.16:4736 , piece_id=5IQAPSUW4KRENWHTIZGI6WPL6RVI3QWQD7XHWJDJKKXF37WKZ66A, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.974887994 (577h44m5.845971039s) [noise=true , address=77.90.31.10:4520 , piece_id=NAAAMF7T3IL44CM3OE56AGEPJWLAPTBCQYR4YT4F3DPJUAFX4RWQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.975052257 (577h44m5.846135302s) [noise=true , address=145.79.227.41:4087 , piece_id=4GNGN4HH5LKBACQSZRQ7F5O5AFE6X2GXNL2P3SIIIOSJES26WH6Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.976593822 (577h44m5.847676867s) [noise=true , address=145.79.227.11:4732 , piece_id=UNFYRITYZO6EFWZ3JJMDCBUNTE5YIZMKDQBMFCW7FZAX7XN7YYOQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.977910088 (577h44m5.848993133s) [noise=true , address=77.90.31.11:4556 , piece_id=DVW7GMDOIABFEMKYP4VLSHWJZZPT3OJ2FGJDCPJ546SPSJPKVAZQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.986389675 (577h44m5.85747272s) [noise=true , address=145.79.227.42:4123 , piece_id=PB4YX5DJU4DZ3EG5OEH44KW225HH2JUJMF3RMEI36JVKGBCUR73Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.989969302 (577h44m5.861052347s) [noise=true , address=145.79.227.32:4836 , piece_id=PZ5A7VLNZBJKYPHVGTIYKDQ5EJEQM7MGYTW6D5IQHSWCYKVHZX4A, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:07.685756914 (577h44m7.556839949s) [upload=506457]: Next returned nil segment Dec 29 17:13:07.685770760 (577h44m7.556853795s) [upload=506457]: Waiting for error group managing segments... Dec 29 17:13:07.968948863 (577h44m7.840031908s) [noise=true , address=145.79.227.11:4732 , piece_id=UNFYRITYZO6EFWZ3JJMDCBUNTE5YIZMKDQBMFCW7FZAX7XN7YYOQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.973726742 (577h44m7.844809777s) [noise=true , address=145.79.227.41:4087 , piece_id=4GNGN4HH5LKBACQSZRQ7F5O5AFE6X2GXNL2P3SIIIOSJES26WH6Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.974384945 (577h44m7.84546799s) [noise=true , address=145.79.227.32:4836 , piece_id=PZ5A7VLNZBJKYPHVGTIYKDQ5EJEQM7MGYTW6D5IQHSWCYKVHZX4A, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.975980733 (577h44m7.847063768s) [noise=true , address=145.79.227.8:4329 , piece_id=LELWF5PB43OACEXDW3HGD7L6CWP3SV7VRHSAYNTGA35DZFKZZDZA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.989921909 (577h44m7.861004944s) [noise=true , address=145.79.227.45:4229 , piece_id=IBBXISAUSWPD6PDBWXYBYRUNZURFLIUTFJRAQJNEXSMKEQEVPR5Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.998113787 (577h44m7.869196822s) [noise=true , address=145.79.227.15:4874 , piece_id=66YNLCBU5Y3KN3X3DDIKWSDDKZY7556DG3U2IVMCXA6VONFNBQSA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.998447743 (577h44m7.869530778s) [noise=true , address=145.79.227.38:4977 , piece_id=5W3CPOBK3BQ6EEWNTO47BHRFEJQSHM7OGM234FUPJ5MYL5R2JKZA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.009744304 (577h44m7.880827339s) [noise=true , address=145.79.227.42:4123 , piece_id=PB4YX5DJU4DZ3EG5OEH44KW225HH2JUJMF3RMEI36JVKGBCUR73Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.151170530 (577h44m8.022253575s) [noise=true , address=77.90.31.16:4736 , piece_id=5IQAPSUW4KRENWHTIZGI6WPL6RVI3QWQD7XHWJDJKKXF37WKZ66A, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.162278511 (577h44m8.033361556s) [noise=true , address=77.90.31.10:4520 , piece_id=NAAAMF7T3IL44CM3OE56AGEPJWLAPTBCQYR4YT4F3DPJUAFX4RWQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.188052118 (577h44m8.059135173s) [noise=true , address=77.90.31.11:4556 , piece_id=DVW7GMDOIABFEMKYP4VLSHWJZZPT3OJ2FGJDCPJ546SPSJPKVAZQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.209634986 (577h44m8.080718031s) [noise=true , address=145.79.227.12:4765 , piece_id=NLNBMVDR6KIS45K2CGRPYQOCJOXA2B4R6R6WK754ASGVEL52S75Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.278822057 (577h44m8.149905102s) [noise=true , address=45.91.116.11:4117 , piece_id=3SKFKEKPXPJH3W4W3TZXMROPPES2I4B2WXFXAWBQM5NHTCDXM6PA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.278847656 (577h44m8.149930701s) [seg_pos={0 0}, upload=506457]: Segment reached optimal threshold of 13 pieces. Dec 29 17:13:08.279089978 (577h44m8.150173023s) [noise=true , address=45.91.116.14:4190 , piece_id=O7KQBNCOJGP5J3FQIJEMJ7LR3PWNFZCDJHM4GCMWXGGO2PRQZ5DQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 17:13:08.279096921 (577h44m8.150179956s) [noise=true , address=45.91.116.14:4190 , piece_id=O7KQBNCOJGP5J3FQIJEMJ7LR3PWNFZCDJHM4GCMWXGGO2PRQZ5DQ, seg_pos={0 0}, upload=506457]: Long tail cancellation due to optimal threshold Dec 29 17:13:08.279239482 (577h44m8.150322517s) [noise=true , address=145.79.227.14:4837 , piece_id=NKPQFIZQYTJ72K2B7UGAIKM5N6JY6YQ7R22KMERFGUOUTBG5STEA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:39686->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:39686->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39686->145.79.227.14:4837: use of closed network connection Dec 29 17:13:08.279244633 (577h44m8.150327668s) [noise=true , address=145.79.227.14:4837 , piece_id=NKPQFIZQYTJ72K2B7UGAIKM5N6JY6YQ7R22KMERFGUOUTBG5STEA, seg_pos={0 0}, upload=506457]: Long tail cancellation due to optimal threshold Dec 29 17:13:08.279259120 (577h44m8.150342155s) [noise=true , address=77.90.31.13:4626 , piece_id=JH2MNAHTSRVOTJLZYH3ZJRSCGAYANO6SK5EODZ6GOVGXY3ETAKHA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:40370->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:40370->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40370->77.90.31.13:4626: use of closed network connection Dec 29 17:13:08.279277354 (577h44m8.150360389s) [noise=true , address=77.90.31.13:4626 , piece_id=JH2MNAHTSRVOTJLZYH3ZJRSCGAYANO6SK5EODZ6GOVGXY3ETAKHA, seg_pos={0 0}, upload=506457]: Long tail cancellation due to optimal threshold Dec 29 17:13:08.279324294 (577h44m8.150407329s) [seg_pos={0 0}, upload=506457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:13:08.279436387 (577h44m8.150519422s) [upload=506457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:13:08.279443200 (577h44m8.150526235s) [upload=506457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:13:08.577230152 (577h44m8.448313197s) [upload=506457]: Done uploading segments... Dec 29 17:13:08.577256682 (577h44m8.448339727s) [upload=506457]: Upload finished. err: <nil> Duration: 6.027790485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.......X......................X........X.............] duration: 42.461905664s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:13:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 42.461944228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...XXXX.XX] duration: 30.005668902s trace: https://jaeger.dev.storj.io/trace/2a0c7ab311525271 FAIL (2025-12-29 17:13:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 3SKFKEKPXPJH3W4W3TZXMROPPES2I4B2WXFXAWBQM5NHTCDXM6PA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: NAAAMF7T3IL44CM3OE56AGEPJWLAPTBCQYR4YT4F3DPJUAFX4RWQ) context deadline exceeded ======LOGS====== 5.583641395s) [upload=506457]: Uploading segments... Dec 29 17:13:05.712739534 (577h44m5.583822579s) [upload=506457]: Got next segment. Inline: false Dec 29 17:13:05.712768079 (577h44m5.583851124s) [upload=506457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:13:05.712772197 (577h44m5.583855232s) [upload=506457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:13:05.969718230 (577h44m5.840801305s) [seg_pos={0 0}, upload=506457]: Begin upload segment... Dec 29 17:13:05.969790878 (577h44m5.840873923s) [seg_pos={0 0}, upload=506457]: Done begin upload segment. Dec 29 17:13:05.969842436 (577h44m5.840925471s) [noise=true , address=45.91.116.11:4117 , piece_id=3SKFKEKPXPJH3W4W3TZXMROPPES2I4B2WXFXAWBQM5NHTCDXM6PA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.969873776 (577h44m5.840956821s) [noise=true , address=45.91.116.14:4190 , piece_id=O7KQBNCOJGP5J3FQIJEMJ7LR3PWNFZCDJHM4GCMWXGGO2PRQZ5DQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.969948157 (577h44m5.841031202s) [noise=true , address=145.79.227.38:4977 , piece_id=5W3CPOBK3BQ6EEWNTO47BHRFEJQSHM7OGM234FUPJ5MYL5R2JKZA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.970407653 (577h44m5.841490688s) [noise=true , address=77.90.31.13:4626 , piece_id=JH2MNAHTSRVOTJLZYH3ZJRSCGAYANO6SK5EODZ6GOVGXY3ETAKHA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.970749062 (577h44m5.841832107s) [noise=true , address=145.79.227.45:4229 , piece_id=IBBXISAUSWPD6PDBWXYBYRUNZURFLIUTFJRAQJNEXSMKEQEVPR5Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.971158402 (577h44m5.842241447s) [noise=true , address=145.79.227.12:4765 , piece_id=NLNBMVDR6KIS45K2CGRPYQOCJOXA2B4R6R6WK754ASGVEL52S75Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.971514339 (577h44m5.842597385s) [noise=true , address=145.79.227.15:4874 , piece_id=66YNLCBU5Y3KN3X3DDIKWSDDKZY7556DG3U2IVMCXA6VONFNBQSA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.972152135 (577h44m5.84323518s) [noise=true , address=145.79.227.14:4837 , piece_id=NKPQFIZQYTJ72K2B7UGAIKM5N6JY6YQ7R22KMERFGUOUTBG5STEA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.972796942 (577h44m5.843879987s) [noise=true , address=145.79.227.8:4329 , piece_id=LELWF5PB43OACEXDW3HGD7L6CWP3SV7VRHSAYNTGA35DZFKZZDZA, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.973446499 (577h44m5.844529544s) [noise=true , address=77.90.31.16:4736 , piece_id=5IQAPSUW4KRENWHTIZGI6WPL6RVI3QWQD7XHWJDJKKXF37WKZ66A, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.974887994 (577h44m5.845971039s) [noise=true , address=77.90.31.10:4520 , piece_id=NAAAMF7T3IL44CM3OE56AGEPJWLAPTBCQYR4YT4F3DPJUAFX4RWQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.975052257 (577h44m5.846135302s) [noise=true , address=145.79.227.41:4087 , piece_id=4GNGN4HH5LKBACQSZRQ7F5O5AFE6X2GXNL2P3SIIIOSJES26WH6Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.976593822 (577h44m5.847676867s) [noise=true , address=145.79.227.11:4732 , piece_id=UNFYRITYZO6EFWZ3JJMDCBUNTE5YIZMKDQBMFCW7FZAX7XN7YYOQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.977910088 (577h44m5.848993133s) [noise=true , address=77.90.31.11:4556 , piece_id=DVW7GMDOIABFEMKYP4VLSHWJZZPT3OJ2FGJDCPJ546SPSJPKVAZQ, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.986389675 (577h44m5.85747272s) [noise=true , address=145.79.227.42:4123 , piece_id=PB4YX5DJU4DZ3EG5OEH44KW225HH2JUJMF3RMEI36JVKGBCUR73Q, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:05.989969302 (577h44m5.861052347s) [noise=true , address=145.79.227.32:4836 , piece_id=PZ5A7VLNZBJKYPHVGTIYKDQ5EJEQM7MGYTW6D5IQHSWCYKVHZX4A, seg_pos={0 0}, upload=506457]: Uploading piece... Dec 29 17:13:07.685756914 (577h44m7.556839949s) [upload=506457]: Next returned nil segment Dec 29 17:13:07.685770760 (577h44m7.556853795s) [upload=506457]: Waiting for error group managing segments... Dec 29 17:13:07.968948863 (577h44m7.840031908s) [noise=true , address=145.79.227.11:4732 , piece_id=UNFYRITYZO6EFWZ3JJMDCBUNTE5YIZMKDQBMFCW7FZAX7XN7YYOQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.973726742 (577h44m7.844809777s) [noise=true , address=145.79.227.41:4087 , piece_id=4GNGN4HH5LKBACQSZRQ7F5O5AFE6X2GXNL2P3SIIIOSJES26WH6Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.974384945 (577h44m7.84546799s) [noise=true , address=145.79.227.32:4836 , piece_id=PZ5A7VLNZBJKYPHVGTIYKDQ5EJEQM7MGYTW6D5IQHSWCYKVHZX4A, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.975980733 (577h44m7.847063768s) [noise=true , address=145.79.227.8:4329 , piece_id=LELWF5PB43OACEXDW3HGD7L6CWP3SV7VRHSAYNTGA35DZFKZZDZA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.989921909 (577h44m7.861004944s) [noise=true , address=145.79.227.45:4229 , piece_id=IBBXISAUSWPD6PDBWXYBYRUNZURFLIUTFJRAQJNEXSMKEQEVPR5Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.998113787 (577h44m7.869196822s) [noise=true , address=145.79.227.15:4874 , piece_id=66YNLCBU5Y3KN3X3DDIKWSDDKZY7556DG3U2IVMCXA6VONFNBQSA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:07.998447743 (577h44m7.869530778s) [noise=true , address=145.79.227.38:4977 , piece_id=5W3CPOBK3BQ6EEWNTO47BHRFEJQSHM7OGM234FUPJ5MYL5R2JKZA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.009744304 (577h44m7.880827339s) [noise=true , address=145.79.227.42:4123 , piece_id=PB4YX5DJU4DZ3EG5OEH44KW225HH2JUJMF3RMEI36JVKGBCUR73Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.151170530 (577h44m8.022253575s) [noise=true , address=77.90.31.16:4736 , piece_id=5IQAPSUW4KRENWHTIZGI6WPL6RVI3QWQD7XHWJDJKKXF37WKZ66A, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.162278511 (577h44m8.033361556s) [noise=true , address=77.90.31.10:4520 , piece_id=NAAAMF7T3IL44CM3OE56AGEPJWLAPTBCQYR4YT4F3DPJUAFX4RWQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.188052118 (577h44m8.059135173s) [noise=true , address=77.90.31.11:4556 , piece_id=DVW7GMDOIABFEMKYP4VLSHWJZZPT3OJ2FGJDCPJ546SPSJPKVAZQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.209634986 (577h44m8.080718031s) [noise=true , address=145.79.227.12:4765 , piece_id=NLNBMVDR6KIS45K2CGRPYQOCJOXA2B4R6R6WK754ASGVEL52S75Q, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.278822057 (577h44m8.149905102s) [noise=true , address=45.91.116.11:4117 , piece_id=3SKFKEKPXPJH3W4W3TZXMROPPES2I4B2WXFXAWBQM5NHTCDXM6PA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: <nil> Dec 29 17:13:08.278847656 (577h44m8.149930701s) [seg_pos={0 0}, upload=506457]: Segment reached optimal threshold of 13 pieces. Dec 29 17:13:08.279089978 (577h44m8.150173023s) [noise=true , address=45.91.116.14:4190 , piece_id=O7KQBNCOJGP5J3FQIJEMJ7LR3PWNFZCDJHM4GCMWXGGO2PRQZ5DQ, seg_pos={0 0}, upload=506457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 17:13:08.279096921 (577h44m8.150179956s) [noise=true , address=45.91.116.14:4190 , piece_id=O7KQBNCOJGP5J3FQIJEMJ7LR3PWNFZCDJHM4GCMWXGGO2PRQZ5DQ, seg_pos={0 0}, upload=506457]: Long tail cancellation due to optimal threshold Dec 29 17:13:08.279239482 (577h44m8.150322517s) [noise=true , address=145.79.227.14:4837 , piece_id=NKPQFIZQYTJ72K2B7UGAIKM5N6JY6YQ7R22KMERFGUOUTBG5STEA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:39686->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:39686->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39686->145.79.227.14:4837: use of closed network connection Dec 29 17:13:08.279244633 (577h44m8.150327668s) [noise=true , address=145.79.227.14:4837 , piece_id=NKPQFIZQYTJ72K2B7UGAIKM5N6JY6YQ7R22KMERFGUOUTBG5STEA, seg_pos={0 0}, upload=506457]: Long tail cancellation due to optimal threshold Dec 29 17:13:08.279259120 (577h44m8.150342155s) [noise=true , address=77.90.31.13:4626 , piece_id=JH2MNAHTSRVOTJLZYH3ZJRSCGAYANO6SK5EODZ6GOVGXY3ETAKHA, seg_pos={0 0}, upload=506457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:40370->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:40370->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40370->77.90.31.13:4626: use of closed network connection Dec 29 17:13:08.279277354 (577h44m8.150360389s) [noise=true , address=77.90.31.13:4626 , piece_id=JH2MNAHTSRVOTJLZYH3ZJRSCGAYANO6SK5EODZ6GOVGXY3ETAKHA, seg_pos={0 0}, upload=506457]: Long tail cancellation due to optimal threshold Dec 29 17:13:08.279324294 (577h44m8.150407329s) [seg_pos={0 0}, upload=506457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:13:08.279436387 (577h44m8.150519422s) [upload=506457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:13:08.279443200 (577h44m8.150526235s) [upload=506457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:13:08.577230152 (577h44m8.448313197s) [upload=506457]: Done uploading segments... Dec 29 17:13:08.577256682 (577h44m8.448339727s) [upload=506457]: Upload finished. err: <nil> Duration: 30.005695964s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.XXXX..X] duration: 5.00206349s trace: https://jaeger.dev.storj.io/trace/241ad6da60babde8 FAIL (2025-12-29 17:13:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002077056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.......X......................X........X............] duration: 26.295850082s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:13:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 26.295876544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...XXXX.X] duration: 30.003426376s trace: https://jaeger.dev.storj.io/trace/35a9ddface009d9f FAIL (2025-12-29 17:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17:12:15.290663350 (577h43m15.161746415s) [upload=506446]: Starting upload Dec 29 17:12:15.290765284 (577h43m15.161848359s) [upload=506446]: Done starting upload Dec 29 17:12:15.290884261 (577h43m15.161967316s) [upload=506446]: Uploading segments... Dec 29 17:12:15.291038414 (577h43m15.162121469s) [upload=506446]: Got next segment. Inline: false Dec 29 17:12:15.291089290 (577h43m15.162172335s) [upload=506446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:12:15.291094882 (577h43m15.162177927s) [upload=506446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:12:15.541135045 (577h43m15.41221808s) [seg_pos={0 0}, upload=506446]: Begin upload segment... Dec 29 17:12:15.541183037 (577h43m15.412266072s) [seg_pos={0 0}, upload=506446]: Done begin upload segment. Dec 29 17:12:15.541208856 (577h43m15.412291891s) [noise=true , address=77.90.31.14:4663 , piece_id=UMTNN66WGI4RDQIVPKXFKUYAGRKDN5U2VCWWBYESQUTL32MKJOIQ, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.541288578 (577h43m15.412371613s) [noise=true , address=145.79.227.37:4566 , piece_id=JZKDQRS3M63ZFBC2TAMJT7OS4NTWV2L6QMPUV2ZKMZ4MNA4WUD7Q, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.542147052 (577h43m15.413230097s) [noise=true , address=145.79.227.28:4726 , piece_id=NV3WPYPHTDER4KH42V64FP27O5FGLUOTSE6NJXFXNJCN75YQJCAA, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.542575047 (577h43m15.413658092s) [noise=true , address=77.90.31.21:4877 , piece_id=V4DIH2PAMFD7FC7V7EFZDH4KXKVX45AK6I6SHAWJXNRWSQ6ZLSQQ, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.542897060 (577h43m15.413980105s) [noise=true , address=145.79.227.45:4229 , piece_id=UYJXCHNNYRIFQEYQTRAIE27B2DET57JIUBC4PTQEYF2JDTKRHWAQ, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.543232038 (577h43m15.414315084s) [noise=true , address=145.79.227.48:4303 , piece_id=SZ4MMJDWTIND7L5RWGACC57CZV6DRB7F6KWWJKMBFWODUGOGHL7A, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.543726360 (577h43m15.414809395s) [noise=true , address=145.79.227.22:4509 , piece_id=HAW57RGPPXVX77HAXMJUCOJUA2ULMZLOA2E4TUNWDDAARI7EBONQ, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.544179132 (577h43m15.415262167s) [noise=true , address=145.79.227.26:4655 , piece_id=FKM2B7SBQXG3HZX2ZPLZ5MKR57A7XVUFY3J6OZVROBMN7JDMBVCQ, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.548275531 (577h43m15.419358566s) [noise=true , address=45.91.116.5:4298 , piece_id=NDI4BQFMQWQXJRK4LKTFLC5FWQGNPHRRCJK34OO3X6WDQUDYWUEQ, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.552367203 (577h43m15.423450238s) [noise=true , address=145.79.227.38:4979 , piece_id=EVKUY3BAZC72KUNCC6EXSDRLO6DRUC2P7V7ZVG7QRHPDFE5Y2EUA, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.555700580 (577h43m15.426783615s) [noise=true , address=77.90.31.16:4736 , piece_id=B274X75BMYEWBJJ3FWGMSSXNGUX2ZAWQ7KVGT6KO2PHZH5DAIE5Q, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.559320232 (577h43m15.430403277s) [noise=true , address=45.91.116.6:4974 , piece_id=OU2VIZRTKN6H6RXRYVYJMGXJOGXTQOFQHYGUF6S3BQRYAXPXOEKA, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.561011303 (577h43m15.432094338s) [noise=true , address=145.79.227.31:4799 , piece_id=E6XBMYKMZTBEYPZHQQTLAC5OO7R2SJEYVU6W7E7SCI47QY2UXNSA, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.561944189 (577h43m15.433027224s) [noise=true , address=145.79.227.41:4087 , piece_id=G7N47AQ2MT4XJ6L4JBMO67BHAKYHMMGFMDM4KB4FIECAFDLXRKWQ, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.563400773 (577h43m15.434483818s) [noise=true , address=45.91.116.12:4225 , piece_id=4V7M2VA25TVI23NX4G6EHK3CCUROAKAZ3JZDIAT2WKCZUXELQZQA, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:15.564813222 (577h43m15.435896257s) [noise=true , address=145.79.227.4:4186 , piece_id=7DGGAHDFDI53575ATV6UQG2B5CEKZJBLLIUZQIQIOMSKGQCW2RAA, seg_pos={0 0}, upload=506446]: Uploading piece... Dec 29 17:12:17.249470960 (577h43m17.120554005s) [upload=506446]: Next returned nil segment Dec 29 17:12:17.249487923 (577h43m17.120570978s) [upload=506446]: Waiting for error group managing segments... Dec 29 17:12:17.528682200 (577h43m17.399765235s) [noise=true , address=145.79.227.48:4303 , piece_id=SZ4MMJDWTIND7L5RWGACC57CZV6DRB7F6KWWJKMBFWODUGOGHL7A, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.532574361 (577h43m17.403657406s) [noise=true , address=145.79.227.45:4229 , piece_id=UYJXCHNNYRIFQEYQTRAIE27B2DET57JIUBC4PTQEYF2JDTKRHWAQ, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.533970911 (577h43m17.405053956s) [noise=true , address=145.79.227.26:4655 , piece_id=FKM2B7SBQXG3HZX2ZPLZ5MKR57A7XVUFY3J6OZVROBMN7JDMBVCQ, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.544679010 (577h43m17.415762045s) [noise=true , address=145.79.227.22:4509 , piece_id=HAW57RGPPXVX77HAXMJUCOJUA2ULMZLOA2E4TUNWDDAARI7EBONQ, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.549946961 (577h43m17.421030006s) [noise=true , address=145.79.227.28:4726 , piece_id=NV3WPYPHTDER4KH42V64FP27O5FGLUOTSE6NJXFXNJCN75YQJCAA, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.551165090 (577h43m17.422248125s) [noise=true , address=145.79.227.4:4186 , piece_id=7DGGAHDFDI53575ATV6UQG2B5CEKZJBLLIUZQIQIOMSKGQCW2RAA, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.553040642 (577h43m17.424123687s) [noise=true , address=145.79.227.37:4566 , piece_id=JZKDQRS3M63ZFBC2TAMJT7OS4NTWV2L6QMPUV2ZKMZ4MNA4WUD7Q, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.553845174 (577h43m17.424928209s) [noise=true , address=145.79.227.38:4979 , piece_id=EVKUY3BAZC72KUNCC6EXSDRLO6DRUC2P7V7ZVG7QRHPDFE5Y2EUA, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.554301063 (577h43m17.425384098s) [noise=true , address=145.79.227.41:4087 , piece_id=G7N47AQ2MT4XJ6L4JBMO67BHAKYHMMGFMDM4KB4FIECAFDLXRKWQ, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.574312727 (577h43m17.445395773s) [noise=true , address=145.79.227.31:4799 , piece_id=E6XBMYKMZTBEYPZHQQTLAC5OO7R2SJEYVU6W7E7SCI47QY2UXNSA, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.752690165 (577h43m17.62377321s) [noise=true , address=77.90.31.16:4736 , piece_id=B274X75BMYEWBJJ3FWGMSSXNGUX2ZAWQ7KVGT6KO2PHZH5DAIE5Q, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.775389868 (577h43m17.646472913s) [noise=true , address=45.91.116.5:4298 , piece_id=NDI4BQFMQWQXJRK4LKTFLC5FWQGNPHRRCJK34OO3X6WDQUDYWUEQ, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.783192907 (577h43m17.654275972s) [noise=true , address=45.91.116.6:4974 , piece_id=OU2VIZRTKN6H6RXRYVYJMGXJOGXTQOFQHYGUF6S3BQRYAXPXOEKA, seg_pos={0 0}, upload=506446]: Done uploading piece. err: <nil> Dec 29 17:12:17.783223164 (577h43m17.654306199s) [seg_pos={0 0}, upload=506446]: Segment reached optimal threshold of 13 pieces. Dec 29 17:12:17.783832434 (577h43m17.654915469s) [noise=true , address=45.91.116.12:4225 , piece_id=4V7M2VA25TVI23NX4G6EHK3CCUROAKAZ3JZDIAT2WKCZUXELQZQA, seg_pos={0 0}, upload=506446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:42776->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:42776->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42776->45.91.116.12:4225: use of closed network connection Dec 29 17:12:17.783841642 (577h43m17.654924677s) [noise=true , address=45.91.116.12:4225 , piece_id=4V7M2VA25TVI23NX4G6EHK3CCUROAKAZ3JZDIAT2WKCZUXELQZQA, seg_pos={0 0}, upload=506446]: Long tail cancellation due to optimal threshold Dec 29 17:12:17.784096106 (577h43m17.655179151s) [noise=true , address=77.90.31.14:4663 , piece_id=UMTNN66WGI4RDQIVPKXFKUYAGRKDN5U2VCWWBYESQUTL32MKJOIQ, seg_pos={0 0}, upload=506446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 17:12:17.784105705 (577h43m17.65518875s) [noise=true , address=77.90.31.14:4663 , piece_id=UMTNN66WGI4RDQIVPKXFKUYAGRKDN5U2VCWWBYESQUTL32MKJOIQ, seg_pos={0 0}, upload=506446]: Long tail cancellation due to optimal threshold Dec 29 17:12:17.784134128 (577h43m17.655217193s) [noise=true , address=77.90.31.21:4877 , piece_id=V4DIH2PAMFD7FC7V7EFZDH4KXKVX45AK6I6SHAWJXNRWSQ6ZLSQQ, seg_pos={0 0}, upload=506446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 17:12:17.784142415 (577h43m17.65522546s) [noise=true , address=77.90.31.21:4877 , piece_id=V4DIH2PAMFD7FC7V7EFZDH4KXKVX45AK6I6SHAWJXNRWSQ6ZLSQQ, seg_pos={0 0}, upload=506446]: Long tail cancellation due to optimal threshold Dec 29 17:12:17.784160619 (577h43m17.655243664s) [seg_pos={0 0}, upload=506446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:12:17.784248947 (577h43m17.655331983s) [upload=506446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:12:17.784254808 (577h43m17.655337854s) [upload=506446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:12:18.077177406 (577h43m17.948260441s) [upload=506446]: Done uploading segments... Dec 29 17:12:18.077195321 (577h43m17.948278356s) [upload=506446]: Upload finished. err: <nil> Duration: 30.003447917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.......X......................X........X...........] duration: 32.403189533s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:12:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 32.403205944s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.XXXX..] duration: 4.903654467s trace: https://jaeger.dev.storj.io/trace/77c4a6e30aafe1dc FAIL (2025-12-29 17:12:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.903681669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.......X......................X........X..........] duration: 25.415387115s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:12:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 25.415402854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...XXXX.] duration: 6.445480672s trace: https://jaeger.dev.storj.io/trace/54423b5b14a06078 FAIL (2025-12-29 17:11:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 55RZO355NQXRLWEOVBNS7VJ3744ZFCPCWI6JPBYKZMLYKBY3HBHQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: 6U3NPTKWLG3QYQDRYWZUMUYGTNFK7X2GC5EQWJZMMMNPJWKZP2HA) context deadline exceeded ======LOGS====== Dec 29 17:10:24.169857617 (577h41m24.040940663s) [upload=506418]: Starting upload Dec 29 17:10:24.169942629 (577h41m24.041025674s) [upload=506418]: Done starting upload Dec 29 17:10:24.170060885 (577h41m24.04114393s) [upload=506418]: Uploading segments... Dec 29 17:10:24.170211742 (577h41m24.041294787s) [upload=506418]: Got next segment. Inline: false Dec 29 17:10:24.170250575 (577h41m24.041333611s) [upload=506418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:10:24.170255445 (577h41m24.04133848s) [upload=506418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:10:24.430625919 (577h41m24.301708984s) [seg_pos={0 0}, upload=506418]: Begin upload segment... Dec 29 17:10:24.430675473 (577h41m24.301758518s) [seg_pos={0 0}, upload=506418]: Done begin upload segment. Dec 29 17:10:24.430703226 (577h41m24.301786261s) [noise=true , address=45.91.116.14:4190 , piece_id=H3W3XAZCUJTAIDB2P4WMFLZZWNOHWQXA4MMQABRMNE34H62FB22A, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.431461330 (577h41m24.302544375s) [noise=true , address=145.79.227.19:4079 , piece_id=CCUB2JX2G5KITYZJK5OLC5NWRZHMIDAXWD43XDB57O4WALMZ3TQQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.431906647 (577h41m24.302989692s) [noise=true , address=145.79.227.52:4446 , piece_id=NILQEOIIG4OBCFETAW6TFJTEZJATLDMOTIFEHJNEGXYTWC6L74RA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.432236857 (577h41m24.303319892s) [noise=true , address=145.79.227.5:4223 , piece_id=J5LBQKOLHTAYZYZWVGVHTJJPWDMF55JV53CJDB22ZS6KEG4YJ2NQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.432520777 (577h41m24.303603812s) [noise=true , address=77.90.31.14:4661 , piece_id=6U3NPTKWLG3QYQDRYWZUMUYGTNFK7X2GC5EQWJZMMMNPJWKZP2HA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.432930728 (577h41m24.304013763s) [noise=true , address=145.79.227.51:4410 , piece_id=UYBKR7DIPPX7WMJPM42LRWLCBD3B2CPG3RV64ML6T5UNNPOTXNXA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.433217223 (577h41m24.304300258s) [noise=true , address=145.79.227.45:4229 , piece_id=N44MCUBIC7FYCWL33OP67GLJ573IYPPV5CCQATCOQUS3H2XZZFBA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.433481747 (577h41m24.304564792s) [noise=true , address=145.79.227.14:4838 , piece_id=WF5EY4Y4JLZZ2GSCDSO4CGZAJ7Z6WTVNQTYN2NABP4OWZUHZ3QMQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.433759115 (577h41m24.304842151s) [noise=true , address=45.91.116.9:4262 , piece_id=IDEX72JKF3ZBWNE76E3PGN2LUAUOPLLV5TG5JACA63KVJQGIFSMQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.435770314 (577h41m24.306853359s) [noise=true , address=145.79.227.46:4603 , piece_id=VUM763WCUVF6UDNC54USU4DOETKUI6VVBYNWIRUS37NLIO7EOLIQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.436085595 (577h41m24.30716864s) [noise=true , address=145.79.227.9:4366 , piece_id=4KBCF3YH6AMHZWCLLBQFG6OZN5C7FNJTU62OIUUKPPXVOYHB4A4A, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.436375326 (577h41m24.307458361s) [noise=true , address=45.91.116.12:4225 , piece_id=JULJ5LFB744CFUJH6VUJXPMNH53KOMXYPDECSNCLJN4LFSW2BIVQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.436675959 (577h41m24.307759004s) [noise=true , address=77.90.31.13:4626 , piece_id=55RZO355NQXRLWEOVBNS7VJ3744ZFCPCWI6JPBYKZMLYKBY3HBHQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.437044140 (577h41m24.308127195s) [noise=true , address=145.79.227.8:4329 , piece_id=YP6NMSNZQ5DDDXWRE33KMI3CSGIWGS4JFIRTSVHAUCQL2LORAVDQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.437374018 (577h41m24.308457073s) [noise=true , address=145.79.227.31:4799 , piece_id=6UEP2DJ53F3LZIMBQVJ7BDORSOUGSNVBBUG6VGYEME4OTACRIUTA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.438810734 (577h41m24.309893769s) [noise=true , address=77.90.31.15:4700 , piece_id=ZKQW6NPFXKGM7CLI2S34S4G4UUSCMIWLDT6DOUUCYN2O3NDK7X6Q, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:26.064597031 (577h41m25.935680076s) [upload=506418]: Next returned nil segment Dec 29 17:10:26.064614443 (577h41m25.935697499s) [upload=506418]: Waiting for error group managing segments... Dec 29 17:10:26.417719257 (577h41m26.288802303s) [noise=true , address=145.79.227.31:4799 , piece_id=6UEP2DJ53F3LZIMBQVJ7BDORSOUGSNVBBUG6VGYEME4OTACRIUTA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.418925174 (577h41m26.290008209s) [noise=true , address=145.79.227.14:4838 , piece_id=WF5EY4Y4JLZZ2GSCDSO4CGZAJ7Z6WTVNQTYN2NABP4OWZUHZ3QMQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.420522174 (577h41m26.291605219s) [noise=true , address=145.79.227.51:4410 , piece_id=UYBKR7DIPPX7WMJPM42LRWLCBD3B2CPG3RV64ML6T5UNNPOTXNXA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.420762091 (577h41m26.291845136s) [noise=true , address=145.79.227.46:4603 , piece_id=VUM763WCUVF6UDNC54USU4DOETKUI6VVBYNWIRUS37NLIO7EOLIQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.423074525 (577h41m26.29415756s) [noise=true , address=145.79.227.45:4229 , piece_id=N44MCUBIC7FYCWL33OP67GLJ573IYPPV5CCQATCOQUS3H2XZZFBA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.424782818 (577h41m26.295865853s) [noise=true , address=145.79.227.19:4079 , piece_id=CCUB2JX2G5KITYZJK5OLC5NWRZHMIDAXWD43XDB57O4WALMZ3TQQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.425966792 (577h41m26.297049837s) [noise=true , address=145.79.227.5:4223 , piece_id=J5LBQKOLHTAYZYZWVGVHTJJPWDMF55JV53CJDB22ZS6KEG4YJ2NQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.432639098 (577h41m26.303722143s) [noise=true , address=145.79.227.9:4366 , piece_id=4KBCF3YH6AMHZWCLLBQFG6OZN5C7FNJTU62OIUUKPPXVOYHB4A4A, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.444419189 (577h41m26.315502224s) [noise=true , address=145.79.227.8:4329 , piece_id=YP6NMSNZQ5DDDXWRE33KMI3CSGIWGS4JFIRTSVHAUCQL2LORAVDQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.609983747 (577h41m26.481066782s) [noise=true , address=77.90.31.14:4661 , piece_id=6U3NPTKWLG3QYQDRYWZUMUYGTNFK7X2GC5EQWJZMMMNPJWKZP2HA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.711291592 (577h41m26.582374638s) [noise=true , address=45.91.116.14:4190 , piece_id=H3W3XAZCUJTAIDB2P4WMFLZZWNOHWQXA4MMQABRMNE34H62FB22A, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.729265507 (577h41m26.600348542s) [noise=true , address=77.90.31.13:4626 , piece_id=55RZO355NQXRLWEOVBNS7VJ3744ZFCPCWI6JPBYKZMLYKBY3HBHQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.740440616 (577h41m26.611523661s) [noise=true , address=77.90.31.15:4700 , piece_id=ZKQW6NPFXKGM7CLI2S34S4G4UUSCMIWLDT6DOUUCYN2O3NDK7X6Q, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.740467567 (577h41m26.611550613s) [seg_pos={0 0}, upload=506418]: Segment reached optimal threshold of 13 pieces. Dec 29 17:10:26.740768220 (577h41m26.611851275s) [noise=true , address=45.91.116.12:4225 , piece_id=JULJ5LFB744CFUJH6VUJXPMNH53KOMXYPDECSNCLJN4LFSW2BIVQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 17:10:26.740780052 (577h41m26.611863098s) [noise=true , address=45.91.116.12:4225 , piece_id=JULJ5LFB744CFUJH6VUJXPMNH53KOMXYPDECSNCLJN4LFSW2BIVQ, seg_pos={0 0}, upload=506418]: Long tail cancellation due to optimal threshold Dec 29 17:10:26.740821621 (577h41m26.611904667s) [noise=true , address=145.79.227.52:4446 , piece_id=NILQEOIIG4OBCFETAW6TFJTEZJATLDMOTIFEHJNEGXYTWC6L74RA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: expected piece hash; context canceled; EOF Dec 29 17:10:26.740939827 (577h41m26.612023093s) [noise=true , address=145.79.227.52:4446 , piece_id=NILQEOIIG4OBCFETAW6TFJTEZJATLDMOTIFEHJNEGXYTWC6L74RA, seg_pos={0 0}, upload=506418]: Long tail cancellation due to optimal threshold Dec 29 17:10:26.741041591 (577h41m26.612124636s) [noise=true , address=45.91.116.9:4262 , piece_id=IDEX72JKF3ZBWNE76E3PGN2LUAUOPLLV5TG5JACA63KVJQGIFSMQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 17:10:26.741054064 (577h41m26.612137099s) [noise=true , address=45.91.116.9:4262 , piece_id=IDEX72JKF3ZBWNE76E3PGN2LUAUOPLLV5TG5JACA63KVJQGIFSMQ, seg_pos={0 0}, upload=506418]: Long tail cancellation due to optimal threshold Dec 29 17:10:26.741080675 (577h41m26.61216373s) [seg_pos={0 0}, upload=506418]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:10:26.741167871 (577h41m26.612250926s) [upload=506418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:10:26.741174785 (577h41m26.61225782s) [upload=506418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:10:27.040913051 (577h41m26.911996096s) [upload=506418]: Done uploading segments... Dec 29 17:10:27.040941565 (577h41m26.91202461s) [upload=506418]: Upload finished. err: <nil> Duration: 6.445504367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.......X......................X........X.........] duration: 40.400699439s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:11:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 40.400726531s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.XXXX.] duration: 4.839460792s trace: https://jaeger.dev.storj.io/trace/77c4a6e30aafe1dc FAIL (2025-12-29 17:11:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.839474918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.XXXX] duration: 5.002160837s trace: https://jaeger.dev.storj.io/trace/77c4a6e30aafe1dc FAIL (2025-12-29 17:11:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002185964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.......X......................X........X........] duration: 40.046420907s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:10:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 40.046451515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...XXXX] duration: 30.003520807s trace: https://jaeger.dev.storj.io/trace/54423b5b14a06078 FAIL (2025-12-29 17:10:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 55RZO355NQXRLWEOVBNS7VJ3744ZFCPCWI6JPBYKZMLYKBY3HBHQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: 6U3NPTKWLG3QYQDRYWZUMUYGTNFK7X2GC5EQWJZMMMNPJWKZP2HA) context deadline exceeded ======LOGS====== Dec 29 17:10:24.169857617 (577h41m24.040940663s) [upload=506418]: Starting upload Dec 29 17:10:24.169942629 (577h41m24.041025674s) [upload=506418]: Done starting upload Dec 29 17:10:24.170060885 (577h41m24.04114393s) [upload=506418]: Uploading segments... Dec 29 17:10:24.170211742 (577h41m24.041294787s) [upload=506418]: Got next segment. Inline: false Dec 29 17:10:24.170250575 (577h41m24.041333611s) [upload=506418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:10:24.170255445 (577h41m24.04133848s) [upload=506418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:10:24.430625919 (577h41m24.301708984s) [seg_pos={0 0}, upload=506418]: Begin upload segment... Dec 29 17:10:24.430675473 (577h41m24.301758518s) [seg_pos={0 0}, upload=506418]: Done begin upload segment. Dec 29 17:10:24.430703226 (577h41m24.301786261s) [noise=true , address=45.91.116.14:4190 , piece_id=H3W3XAZCUJTAIDB2P4WMFLZZWNOHWQXA4MMQABRMNE34H62FB22A, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.431461330 (577h41m24.302544375s) [noise=true , address=145.79.227.19:4079 , piece_id=CCUB2JX2G5KITYZJK5OLC5NWRZHMIDAXWD43XDB57O4WALMZ3TQQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.431906647 (577h41m24.302989692s) [noise=true , address=145.79.227.52:4446 , piece_id=NILQEOIIG4OBCFETAW6TFJTEZJATLDMOTIFEHJNEGXYTWC6L74RA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.432236857 (577h41m24.303319892s) [noise=true , address=145.79.227.5:4223 , piece_id=J5LBQKOLHTAYZYZWVGVHTJJPWDMF55JV53CJDB22ZS6KEG4YJ2NQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.432520777 (577h41m24.303603812s) [noise=true , address=77.90.31.14:4661 , piece_id=6U3NPTKWLG3QYQDRYWZUMUYGTNFK7X2GC5EQWJZMMMNPJWKZP2HA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.432930728 (577h41m24.304013763s) [noise=true , address=145.79.227.51:4410 , piece_id=UYBKR7DIPPX7WMJPM42LRWLCBD3B2CPG3RV64ML6T5UNNPOTXNXA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.433217223 (577h41m24.304300258s) [noise=true , address=145.79.227.45:4229 , piece_id=N44MCUBIC7FYCWL33OP67GLJ573IYPPV5CCQATCOQUS3H2XZZFBA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.433481747 (577h41m24.304564792s) [noise=true , address=145.79.227.14:4838 , piece_id=WF5EY4Y4JLZZ2GSCDSO4CGZAJ7Z6WTVNQTYN2NABP4OWZUHZ3QMQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.433759115 (577h41m24.304842151s) [noise=true , address=45.91.116.9:4262 , piece_id=IDEX72JKF3ZBWNE76E3PGN2LUAUOPLLV5TG5JACA63KVJQGIFSMQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.435770314 (577h41m24.306853359s) [noise=true , address=145.79.227.46:4603 , piece_id=VUM763WCUVF6UDNC54USU4DOETKUI6VVBYNWIRUS37NLIO7EOLIQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.436085595 (577h41m24.30716864s) [noise=true , address=145.79.227.9:4366 , piece_id=4KBCF3YH6AMHZWCLLBQFG6OZN5C7FNJTU62OIUUKPPXVOYHB4A4A, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.436375326 (577h41m24.307458361s) [noise=true , address=45.91.116.12:4225 , piece_id=JULJ5LFB744CFUJH6VUJXPMNH53KOMXYPDECSNCLJN4LFSW2BIVQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.436675959 (577h41m24.307759004s) [noise=true , address=77.90.31.13:4626 , piece_id=55RZO355NQXRLWEOVBNS7VJ3744ZFCPCWI6JPBYKZMLYKBY3HBHQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.437044140 (577h41m24.308127195s) [noise=true , address=145.79.227.8:4329 , piece_id=YP6NMSNZQ5DDDXWRE33KMI3CSGIWGS4JFIRTSVHAUCQL2LORAVDQ, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.437374018 (577h41m24.308457073s) [noise=true , address=145.79.227.31:4799 , piece_id=6UEP2DJ53F3LZIMBQVJ7BDORSOUGSNVBBUG6VGYEME4OTACRIUTA, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:24.438810734 (577h41m24.309893769s) [noise=true , address=77.90.31.15:4700 , piece_id=ZKQW6NPFXKGM7CLI2S34S4G4UUSCMIWLDT6DOUUCYN2O3NDK7X6Q, seg_pos={0 0}, upload=506418]: Uploading piece... Dec 29 17:10:26.064597031 (577h41m25.935680076s) [upload=506418]: Next returned nil segment Dec 29 17:10:26.064614443 (577h41m25.935697499s) [upload=506418]: Waiting for error group managing segments... Dec 29 17:10:26.417719257 (577h41m26.288802303s) [noise=true , address=145.79.227.31:4799 , piece_id=6UEP2DJ53F3LZIMBQVJ7BDORSOUGSNVBBUG6VGYEME4OTACRIUTA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.418925174 (577h41m26.290008209s) [noise=true , address=145.79.227.14:4838 , piece_id=WF5EY4Y4JLZZ2GSCDSO4CGZAJ7Z6WTVNQTYN2NABP4OWZUHZ3QMQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.420522174 (577h41m26.291605219s) [noise=true , address=145.79.227.51:4410 , piece_id=UYBKR7DIPPX7WMJPM42LRWLCBD3B2CPG3RV64ML6T5UNNPOTXNXA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.420762091 (577h41m26.291845136s) [noise=true , address=145.79.227.46:4603 , piece_id=VUM763WCUVF6UDNC54USU4DOETKUI6VVBYNWIRUS37NLIO7EOLIQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.423074525 (577h41m26.29415756s) [noise=true , address=145.79.227.45:4229 , piece_id=N44MCUBIC7FYCWL33OP67GLJ573IYPPV5CCQATCOQUS3H2XZZFBA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.424782818 (577h41m26.295865853s) [noise=true , address=145.79.227.19:4079 , piece_id=CCUB2JX2G5KITYZJK5OLC5NWRZHMIDAXWD43XDB57O4WALMZ3TQQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.425966792 (577h41m26.297049837s) [noise=true , address=145.79.227.5:4223 , piece_id=J5LBQKOLHTAYZYZWVGVHTJJPWDMF55JV53CJDB22ZS6KEG4YJ2NQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.432639098 (577h41m26.303722143s) [noise=true , address=145.79.227.9:4366 , piece_id=4KBCF3YH6AMHZWCLLBQFG6OZN5C7FNJTU62OIUUKPPXVOYHB4A4A, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.444419189 (577h41m26.315502224s) [noise=true , address=145.79.227.8:4329 , piece_id=YP6NMSNZQ5DDDXWRE33KMI3CSGIWGS4JFIRTSVHAUCQL2LORAVDQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.609983747 (577h41m26.481066782s) [noise=true , address=77.90.31.14:4661 , piece_id=6U3NPTKWLG3QYQDRYWZUMUYGTNFK7X2GC5EQWJZMMMNPJWKZP2HA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.711291592 (577h41m26.582374638s) [noise=true , address=45.91.116.14:4190 , piece_id=H3W3XAZCUJTAIDB2P4WMFLZZWNOHWQXA4MMQABRMNE34H62FB22A, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.729265507 (577h41m26.600348542s) [noise=true , address=77.90.31.13:4626 , piece_id=55RZO355NQXRLWEOVBNS7VJ3744ZFCPCWI6JPBYKZMLYKBY3HBHQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.740440616 (577h41m26.611523661s) [noise=true , address=77.90.31.15:4700 , piece_id=ZKQW6NPFXKGM7CLI2S34S4G4UUSCMIWLDT6DOUUCYN2O3NDK7X6Q, seg_pos={0 0}, upload=506418]: Done uploading piece. err: <nil> Dec 29 17:10:26.740467567 (577h41m26.611550613s) [seg_pos={0 0}, upload=506418]: Segment reached optimal threshold of 13 pieces. Dec 29 17:10:26.740768220 (577h41m26.611851275s) [noise=true , address=45.91.116.12:4225 , piece_id=JULJ5LFB744CFUJH6VUJXPMNH53KOMXYPDECSNCLJN4LFSW2BIVQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 17:10:26.740780052 (577h41m26.611863098s) [noise=true , address=45.91.116.12:4225 , piece_id=JULJ5LFB744CFUJH6VUJXPMNH53KOMXYPDECSNCLJN4LFSW2BIVQ, seg_pos={0 0}, upload=506418]: Long tail cancellation due to optimal threshold Dec 29 17:10:26.740821621 (577h41m26.611904667s) [noise=true , address=145.79.227.52:4446 , piece_id=NILQEOIIG4OBCFETAW6TFJTEZJATLDMOTIFEHJNEGXYTWC6L74RA, seg_pos={0 0}, upload=506418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: expected piece hash; context canceled; EOF Dec 29 17:10:26.740939827 (577h41m26.612023093s) [noise=true , address=145.79.227.52:4446 , piece_id=NILQEOIIG4OBCFETAW6TFJTEZJATLDMOTIFEHJNEGXYTWC6L74RA, seg_pos={0 0}, upload=506418]: Long tail cancellation due to optimal threshold Dec 29 17:10:26.741041591 (577h41m26.612124636s) [noise=true , address=45.91.116.9:4262 , piece_id=IDEX72JKF3ZBWNE76E3PGN2LUAUOPLLV5TG5JACA63KVJQGIFSMQ, seg_pos={0 0}, upload=506418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 17:10:26.741054064 (577h41m26.612137099s) [noise=true , address=45.91.116.9:4262 , piece_id=IDEX72JKF3ZBWNE76E3PGN2LUAUOPLLV5TG5JACA63KVJQGIFSMQ, seg_pos={0 0}, upload=506418]: Long tail cancellation due to optimal threshold Dec 29 17:10:26.741080675 (577h41m26.61216373s) [seg_pos={0 0}, upload=506418]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:10:26.741167871 (577h41m26.612250926s) [upload=506418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:10:26.741174785 (577h41m26.61225782s) [upload=506418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:10:27.040913051 (577h41m26.911996096s) [upload=506418]: Done uploading segments... Dec 29 17:10:27.040941565 (577h41m26.91202461s) [upload=506418]: Upload finished. err: <nil> Duration: 30.003548259s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.XXX] duration: 5.000925067s trace: https://jaeger.dev.storj.io/trace/751c4561cc05fe1b FAIL (2025-12-29 17:10:14 UTC) orbiter warning: context deadline exceeded Duration: 5.000952148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.......X......................X........X.......] duration: 25.743708645s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 25.74372723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [..XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...XXX] duration: 30.004122731s trace: https://jaeger.dev.storj.io/trace/101a678e9cded8b3 FAIL (2025-12-29 17:10:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BWTKH2VMZZ7E55A4FKHDS3WFQ5INP3Q4YPBA2PFSPJNPH7NUHBEA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 7FOHPQGAGWB7ZK2OHO7BWAQ5T3RTEBTM7VPID6LVPMD5LOTDKCUQ) context deadline exceeded ======LOGS====== 9 17:09:36.300942314 (577h40m36.172025359s) [upload=506404]: Starting upload Dec 29 17:09:36.301027427 (577h40m36.172110482s) [upload=506404]: Done starting upload Dec 29 17:09:36.301142726 (577h40m36.172225761s) [upload=506404]: Uploading segments... Dec 29 17:09:36.301396889 (577h40m36.172479924s) [upload=506404]: Got next segment. Inline: false Dec 29 17:09:36.301441926 (577h40m36.172524961s) [upload=506404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:09:36.301447446 (577h40m36.172530491s) [upload=506404]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:09:36.549045229 (577h40m36.420128275s) [seg_pos={0 0}, upload=506404]: Begin upload segment... Dec 29 17:09:36.549103420 (577h40m36.420186465s) [seg_pos={0 0}, upload=506404]: Done begin upload segment. Dec 29 17:09:36.549160028 (577h40m36.420243063s) [noise=true , address=77.90.31.10:4520 , piece_id=HPEWXV6RZ2ED4S7ODGZOEFJS6KMU22T273OC54K65JWCZW3ADEBA, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.549175727 (577h40m36.420258762s) [noise=true , address=77.90.31.21:4877 , piece_id=2IICBTEPRFDR6O72Z2UWUNDSB6QWLX2QQVSD2G6S7PNNMWMC54KQ, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.549679417 (577h40m36.420762452s) [noise=true , address=145.79.227.24:4582 , piece_id=X3FTOPN4OGYZG3WLOA4QXZPBUP6AEE5OV2EMOL5YZDFWOTBTNSSA, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.549700506 (577h40m36.420783541s) [noise=true , address=145.79.227.12:4767 , piece_id=5ERH5KLPFPD4Y6FANPLCIFTNYZBBJBWM6YFAPNBL4M57YI4YL65Q, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.550107732 (577h40m36.421190777s) [noise=true , address=45.91.116.7:4011 , piece_id=7FOHPQGAGWB7ZK2OHO7BWAQ5T3RTEBTM7VPID6LVPMD5LOTDKCUQ, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.550570974 (577h40m36.421654019s) [noise=true , address=77.90.31.16:4735 , piece_id=6VE5EPNBVBDJS5TJKELRQDYP3Y77AFKFC5I76VC4IWDC727DVTJQ, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.550854334 (577h40m36.421937369s) [noise=true , address=77.90.31.14:4661 , piece_id=JJ2ZADYD5Q74ZG3GPD2PZHVME7AUYX46JQ5HXSMOIJYPDKKXTRUQ, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.551108268 (577h40m36.422191303s) [noise=true , address=145.79.227.5:4221 , piece_id=CSH3SKKTB7B4DRP6YQ6SVWZO2Q2YSHY45USFNU7QLHCXW3SEERYA, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.551387119 (577h40m36.422470154s) [noise=true , address=45.91.116.13:4154 , piece_id=CTJP4GP4CPPX6QDBD3COHC2MC3U4IOGCVFJMIWSJJ7JKXN4NE7UA, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.551673754 (577h40m36.422756799s) [noise=true , address=145.79.227.46:4603 , piece_id=5IZ6QZLDSUDRHCWD5FKYDOIPIJCQMA7BXF47TBZDX4BK64ZSSEWA, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.552057284 (577h40m36.42314033s) [noise=true , address=145.79.227.22:4510 , piece_id=PYXSNPBFI5QOMMN3S2DF3XPS4OBT5KG3FU5JXVNW4SVC7QUTMIRA, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.555608827 (577h40m36.426691862s) [noise=true , address=45.91.116.14:4189 , piece_id=VCRZYII7JUEITLIHWRVS4CSZ3U5KDSE6MAH534SM6CQ6EZFLZG5Q, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.556740111 (577h40m36.427823156s) [noise=true , address=145.79.227.9:4366 , piece_id=FP2CY27AUM5BXO5KQZZK5N6SOVW4LPFTYHDKZYXYZLZ33DKQRQ3A, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.560173589 (577h40m36.431256634s) [noise=true , address=145.79.227.41:4087 , piece_id=J2QN7UE5V7WCYO22L3XTG7PI6UPJU6LWZQI3LAVOH6TBYCU3BRUQ, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.561454198 (577h40m36.432537243s) [noise=true , address=145.79.227.4:4186 , piece_id=PNVKQP4JUS4XB3H56JL4NPUQTJEEY6NSZR4UNTLPOIZW74SOGCJQ, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:36.563850601 (577h40m36.434933636s) [noise=true , address=45.91.116.6:4973 , piece_id=BWTKH2VMZZ7E55A4FKHDS3WFQ5INP3Q4YPBA2PFSPJNPH7NUHBEA, seg_pos={0 0}, upload=506404]: Uploading piece... Dec 29 17:09:38.258116705 (577h40m38.12919974s) [upload=506404]: Next returned nil segment Dec 29 17:09:38.258130712 (577h40m38.129213747s) [upload=506404]: Waiting for error group managing segments... Dec 29 17:09:38.539729728 (577h40m38.410812763s) [noise=true , address=145.79.227.5:4221 , piece_id=CSH3SKKTB7B4DRP6YQ6SVWZO2Q2YSHY45USFNU7QLHCXW3SEERYA, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.547895467 (577h40m38.418978512s) [noise=true , address=145.79.227.12:4767 , piece_id=5ERH5KLPFPD4Y6FANPLCIFTNYZBBJBWM6YFAPNBL4M57YI4YL65Q, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.549932756 (577h40m38.421015791s) [noise=true , address=145.79.227.46:4603 , piece_id=5IZ6QZLDSUDRHCWD5FKYDOIPIJCQMA7BXF47TBZDX4BK64ZSSEWA, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.555763548 (577h40m38.426846603s) [noise=true , address=145.79.227.9:4366 , piece_id=FP2CY27AUM5BXO5KQZZK5N6SOVW4LPFTYHDKZYXYZLZ33DKQRQ3A, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.557783484 (577h40m38.42886653s) [noise=true , address=145.79.227.4:4186 , piece_id=PNVKQP4JUS4XB3H56JL4NPUQTJEEY6NSZR4UNTLPOIZW74SOGCJQ, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.558328913 (577h40m38.429411968s) [noise=true , address=145.79.227.41:4087 , piece_id=J2QN7UE5V7WCYO22L3XTG7PI6UPJU6LWZQI3LAVOH6TBYCU3BRUQ, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.566523777 (577h40m38.437606812s) [noise=true , address=145.79.227.24:4582 , piece_id=X3FTOPN4OGYZG3WLOA4QXZPBUP6AEE5OV2EMOL5YZDFWOTBTNSSA, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.570298896 (577h40m38.441381942s) [noise=true , address=145.79.227.22:4510 , piece_id=PYXSNPBFI5QOMMN3S2DF3XPS4OBT5KG3FU5JXVNW4SVC7QUTMIRA, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.725648053 (577h40m38.596731088s) [noise=true , address=77.90.31.10:4520 , piece_id=HPEWXV6RZ2ED4S7ODGZOEFJS6KMU22T273OC54K65JWCZW3ADEBA, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.803470860 (577h40m38.674553905s) [noise=true , address=45.91.116.7:4011 , piece_id=7FOHPQGAGWB7ZK2OHO7BWAQ5T3RTEBTM7VPID6LVPMD5LOTDKCUQ, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.811848652 (577h40m38.682931687s) [noise=true , address=77.90.31.16:4735 , piece_id=6VE5EPNBVBDJS5TJKELRQDYP3Y77AFKFC5I76VC4IWDC727DVTJQ, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.837518341 (577h40m38.708601376s) [noise=true , address=45.91.116.6:4973 , piece_id=BWTKH2VMZZ7E55A4FKHDS3WFQ5INP3Q4YPBA2PFSPJNPH7NUHBEA, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.848572119 (577h40m38.719655154s) [noise=true , address=77.90.31.14:4661 , piece_id=JJ2ZADYD5Q74ZG3GPD2PZHVME7AUYX46JQ5HXSMOIJYPDKKXTRUQ, seg_pos={0 0}, upload=506404]: Done uploading piece. err: <nil> Dec 29 17:09:38.848592418 (577h40m38.719675453s) [seg_pos={0 0}, upload=506404]: Segment reached optimal threshold of 13 pieces. Dec 29 17:09:38.848784444 (577h40m38.719867479s) [noise=true , address=77.90.31.21:4877 , piece_id=2IICBTEPRFDR6O72Z2UWUNDSB6QWLX2QQVSD2G6S7PNNMWMC54KQ, seg_pos={0 0}, upload=506404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 17:09:38.848790726 (577h40m38.719873761s) [noise=true , address=77.90.31.21:4877 , piece_id=2IICBTEPRFDR6O72Z2UWUNDSB6QWLX2QQVSD2G6S7PNNMWMC54KQ, seg_pos={0 0}, upload=506404]: Long tail cancellation due to optimal threshold Dec 29 17:09:38.848990476 (577h40m38.720073511s) [noise=true , address=45.91.116.14:4189 , piece_id=VCRZYII7JUEITLIHWRVS4CSZ3U5KDSE6MAH534SM6CQ6EZFLZG5Q, seg_pos={0 0}, upload=506404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 29 17:09:38.848996688 (577h40m38.720079733s) [noise=true , address=45.91.116.14:4189 , piece_id=VCRZYII7JUEITLIHWRVS4CSZ3U5KDSE6MAH534SM6CQ6EZFLZG5Q, seg_pos={0 0}, upload=506404]: Long tail cancellation due to optimal threshold Dec 29 17:09:38.849043256 (577h40m38.720126291s) [noise=true , address=45.91.116.13:4154 , piece_id=CTJP4GP4CPPX6QDBD3COHC2MC3U4IOGCVFJMIWSJJ7JKXN4NE7UA, seg_pos={0 0}, upload=506404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:40148->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:40148->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40148->45.91.116.13:4154: use of closed network connection Dec 29 17:09:38.849075237 (577h40m38.720158272s) [noise=true , address=45.91.116.13:4154 , piece_id=CTJP4GP4CPPX6QDBD3COHC2MC3U4IOGCVFJMIWSJJ7JKXN4NE7UA, seg_pos={0 0}, upload=506404]: Long tail cancellation due to optimal threshold Dec 29 17:09:38.849104113 (577h40m38.720187148s) [seg_pos={0 0}, upload=506404]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:09:38.849224341 (577h40m38.720307376s) [upload=506404]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:09:38.849229551 (577h40m38.720312586s) [upload=506404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:09:39.144239915 (577h40m39.01532296s) [upload=506404]: Done uploading segments... Dec 29 17:09:39.144260935 (577h40m39.01534397s) [upload=506404]: Upload finished. err: <nil> Duration: 30.004149312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.......X......................X........X......] duration: 36.68368305s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:09:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 36.68369955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [X.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.XX] duration: 5.001361358s trace: https://jaeger.dev.storj.io/trace/2b627a9678d03a5a FAIL (2025-12-29 17:09:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001410911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...XX] duration: 30.00173727s trace: https://jaeger.dev.storj.io/trace/680625d1df1c5f35 FAIL (2025-12-29 17:09:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: WLE6GNM3IASMMMBQHDSBSMQVVF4CRUL2ML3FREVWZPRVXDUKJE2Q) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: EJX57IRMRN36VE4NHNZWCYFBYILIPG6PK4WDPWBQG2IUMQ2QFSYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DA27DFR43G2HDNXG6SNJHS7WPNCBYUCP7BXFDQK4TOPDUIMC67BQ) context deadline exceeded ======LOGS====== 29 17:08:51.040966320 (577h39m50.912049365s) [upload=506395]: Starting upload Dec 29 17:08:51.041069897 (577h39m50.912152932s) [upload=506395]: Done starting upload Dec 29 17:08:51.041152855 (577h39m50.91223589s) [upload=506395]: Uploading segments... Dec 29 17:08:51.041432066 (577h39m50.912515101s) [upload=506395]: Got next segment. Inline: false Dec 29 17:08:51.041566052 (577h39m50.912649087s) [upload=506395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:08:51.041575721 (577h39m50.912658756s) [upload=506395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:08:51.284226133 (577h39m51.155309188s) [seg_pos={0 0}, upload=506395]: Begin upload segment... Dec 29 17:08:51.284295084 (577h39m51.155378119s) [seg_pos={0 0}, upload=506395]: Done begin upload segment. Dec 29 17:08:51.284330992 (577h39m51.155414037s) [noise=true , address=145.79.227.51:4410 , piece_id=GAOF6YOJDMUSZV45YO6IQEQSMOLW6LS2QAD57O5SIBM6AUSIHBYA, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.284341022 (577h39m51.155424067s) [noise=true , address=145.79.227.17:4970 , piece_id=6FIHVD6XHOXG6EVDWTBF75OQ73XBF5VSCYDZEGGHXWFNMK34RHFQ, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.285142739 (577h39m51.156225784s) [noise=true , address=145.79.227.50:4374 , piece_id=EKK6GEXC2AT2B7ZE7AKKU2XREJI6LQE4H5BYKPKVA545KGM7E6JA, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.285717563 (577h39m51.156800598s) [noise=true , address=45.91.116.10:4084 , piece_id=WLE6GNM3IASMMMBQHDSBSMQVVF4CRUL2ML3FREVWZPRVXDUKJE2Q, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.286297718 (577h39m51.157380753s) [noise=true , address=45.91.116.5:4299 , piece_id=D3YUYDLGT5GMNKBPN2LSCLMCQE4INJNWKDJWLZC7UPSTS7H6PCUQ, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.286828147 (577h39m51.157911193s) [noise=true , address=77.90.31.19:4841 , piece_id=EJX57IRMRN36VE4NHNZWCYFBYILIPG6PK4WDPWBQG2IUMQ2QFSYA, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.287478015 (577h39m51.15856107s) [noise=true , address=45.91.116.4:4937 , piece_id=2RVGBQXJ5XB2IMJ22AI3PDZY5ZXRRF74BBQP55UYNC6W4MEX7OHQ, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.287994860 (577h39m51.159077895s) [noise=true , address=145.79.227.19:4077 , piece_id=WAXPRLMO5YOS3VZ7PA6BJ2XIN7N5D4DEKCJYGE2WCHMGOGPSAXTQ, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.288601625 (577h39m51.15968467s) [noise=true , address=45.91.116.8:4047 , piece_id=DI7P332ZA5KNUU5OMHVMVVUPR7J5YTZLBBIFGQTF7ZYQ5IMPD5GA, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.289272933 (577h39m51.160355978s) [noise=true , address=45.91.116.6:4975 , piece_id=DA27DFR43G2HDNXG6SNJHS7WPNCBYUCP7BXFDQK4TOPDUIMC67BQ, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.291541413 (577h39m51.162624468s) [noise=true , address=145.79.227.39:4015 , piece_id=SMCYYDEOLZJJVFYCWQ7L66TN7U2KMTRYFNW3FMYNZGGJMN6K5NRQ, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.294221335 (577h39m51.16530438s) [noise=true , address=145.79.227.5:4223 , piece_id=CU7UMAWC7QGZ54JB2AXY3OFNSI3RXVR7OZT56CB3A6J6OU7HN2XA, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.296341061 (577h39m51.167424106s) [noise=true , address=145.79.227.11:4729 , piece_id=Q5SOYLVXMTMEZ23UPZLAT24D5WQWVVZ36PGI6K4JAIYJSJUSV52Q, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.298715963 (577h39m51.169799008s) [noise=true , address=77.90.31.10:4519 , piece_id=62VBDQ4R7PIEFYRQNPOSB753O2PZYMGIX6ZZCMCC4XUDRNA44DUQ, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.299070989 (577h39m51.170154024s) [noise=true , address=145.79.227.22:4509 , piece_id=64CAFOWMXCT56QM5WYVZGOPNZW43SOIXJ6GWW7DDVQEQYS2ITE6A, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:51.300459704 (577h39m51.171542749s) [noise=true , address=145.79.227.12:4765 , piece_id=IIWIZRYOXMEVHAOXPDY6DR6E2U2OQOOHEMXZAB6JGGKOX7NL7TQA, seg_pos={0 0}, upload=506395]: Uploading piece... Dec 29 17:08:53.001374639 (577h39m52.872457674s) [upload=506395]: Next returned nil segment Dec 29 17:08:53.001398093 (577h39m52.872481138s) [upload=506395]: Waiting for error group managing segments... Dec 29 17:08:53.281489108 (577h39m53.152572143s) [noise=true , address=145.79.227.12:4765 , piece_id=IIWIZRYOXMEVHAOXPDY6DR6E2U2OQOOHEMXZAB6JGGKOX7NL7TQA, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.285135511 (577h39m53.156218546s) [noise=true , address=145.79.227.51:4410 , piece_id=GAOF6YOJDMUSZV45YO6IQEQSMOLW6LS2QAD57O5SIBM6AUSIHBYA, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.286431199 (577h39m53.157514234s) [noise=true , address=145.79.227.19:4077 , piece_id=WAXPRLMO5YOS3VZ7PA6BJ2XIN7N5D4DEKCJYGE2WCHMGOGPSAXTQ, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.287143354 (577h39m53.158226389s) [noise=true , address=145.79.227.39:4015 , piece_id=SMCYYDEOLZJJVFYCWQ7L66TN7U2KMTRYFNW3FMYNZGGJMN6K5NRQ, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.289076296 (577h39m53.160159341s) [noise=true , address=145.79.227.11:4729 , piece_id=Q5SOYLVXMTMEZ23UPZLAT24D5WQWVVZ36PGI6K4JAIYJSJUSV52Q, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.292285146 (577h39m53.163368191s) [noise=true , address=145.79.227.17:4970 , piece_id=6FIHVD6XHOXG6EVDWTBF75OQ73XBF5VSCYDZEGGHXWFNMK34RHFQ, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.297436584 (577h39m53.168519619s) [noise=true , address=145.79.227.22:4509 , piece_id=64CAFOWMXCT56QM5WYVZGOPNZW43SOIXJ6GWW7DDVQEQYS2ITE6A, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.307269929 (577h39m53.178352964s) [noise=true , address=145.79.227.50:4374 , piece_id=EKK6GEXC2AT2B7ZE7AKKU2XREJI6LQE4H5BYKPKVA545KGM7E6JA, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.522759042 (577h39m53.393842077s) [noise=true , address=45.91.116.6:4975 , piece_id=DA27DFR43G2HDNXG6SNJHS7WPNCBYUCP7BXFDQK4TOPDUIMC67BQ, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.528315502 (577h39m53.399398537s) [noise=true , address=77.90.31.19:4841 , piece_id=EJX57IRMRN36VE4NHNZWCYFBYILIPG6PK4WDPWBQG2IUMQ2QFSYA, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.576191174 (577h39m53.447274219s) [noise=true , address=77.90.31.10:4519 , piece_id=62VBDQ4R7PIEFYRQNPOSB753O2PZYMGIX6ZZCMCC4XUDRNA44DUQ, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.649475849 (577h39m53.520558884s) [noise=true , address=145.79.227.5:4223 , piece_id=CU7UMAWC7QGZ54JB2AXY3OFNSI3RXVR7OZT56CB3A6J6OU7HN2XA, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.655303466 (577h39m53.526386501s) [noise=true , address=45.91.116.10:4084 , piece_id=WLE6GNM3IASMMMBQHDSBSMQVVF4CRUL2ML3FREVWZPRVXDUKJE2Q, seg_pos={0 0}, upload=506395]: Done uploading piece. err: <nil> Dec 29 17:08:53.655321851 (577h39m53.526404886s) [seg_pos={0 0}, upload=506395]: Segment reached optimal threshold of 13 pieces. Dec 29 17:08:53.655523886 (577h39m53.526606921s) [noise=true , address=45.91.116.5:4299 , piece_id=D3YUYDLGT5GMNKBPN2LSCLMCQE4INJNWKDJWLZC7UPSTS7H6PCUQ, seg_pos={0 0}, upload=506395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 29 17:08:53.655530238 (577h39m53.526613273s) [noise=true , address=45.91.116.5:4299 , piece_id=D3YUYDLGT5GMNKBPN2LSCLMCQE4INJNWKDJWLZC7UPSTS7H6PCUQ, seg_pos={0 0}, upload=506395]: Long tail cancellation due to optimal threshold Dec 29 17:08:53.655536129 (577h39m53.526619164s) [noise=true , address=45.91.116.8:4047 , piece_id=DI7P332ZA5KNUU5OMHVMVVUPR7J5YTZLBBIFGQTF7ZYQ5IMPD5GA, seg_pos={0 0}, upload=506395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:47646->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:47646->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47646->45.91.116.8:4047: use of closed network connection Dec 29 17:08:53.655544114 (577h39m53.526627159s) [noise=true , address=45.91.116.8:4047 , piece_id=DI7P332ZA5KNUU5OMHVMVVUPR7J5YTZLBBIFGQTF7ZYQ5IMPD5GA, seg_pos={0 0}, upload=506395]: Long tail cancellation due to optimal threshold Dec 29 17:08:53.655717835 (577h39m53.52680088s) [noise=true , address=45.91.116.4:4937 , piece_id=2RVGBQXJ5XB2IMJ22AI3PDZY5ZXRRF74BBQP55UYNC6W4MEX7OHQ, seg_pos={0 0}, upload=506395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 17:08:53.655725820 (577h39m53.526808855s) [noise=true , address=45.91.116.4:4937 , piece_id=2RVGBQXJ5XB2IMJ22AI3PDZY5ZXRRF74BBQP55UYNC6W4MEX7OHQ, seg_pos={0 0}, upload=506395]: Long tail cancellation due to optimal threshold Dec 29 17:08:53.655740238 (577h39m53.526823283s) [seg_pos={0 0}, upload=506395]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:08:53.655807206 (577h39m53.526890251s) [upload=506395]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:08:53.655811754 (577h39m53.526894799s) [upload=506395]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:08:53.948751465 (577h39m53.81983451s) [upload=506395]: Done uploading segments... Dec 29 17:08:53.948771233 (577h39m53.819854278s) [upload=506395]: Upload finished. err: <nil> Duration: 30.001760996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.......X......................X........X.....] duration: 33.800735916s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:09:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 33.800750014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.X] duration: 5.001709396s trace: https://jaeger.dev.storj.io/trace/4e9df0d2f5f03bed FAIL (2025-12-29 17:08:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001723663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.......X......................X........X....] duration: 40.96535859s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:08:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 40.965383607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...X] duration: 30.003228678s trace: https://jaeger.dev.storj.io/trace/577fc544476cf76a FAIL (2025-12-29 17:08:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 17:07:47.961282961 (577h38m47.832366007s) [upload=506381]: Got next segment. Inline: false Dec 29 17:07:47.961322978 (577h38m47.832406013s) [upload=506381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:07:47.961327586 (577h38m47.832410621s) [upload=506381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:07:48.212119449 (577h38m48.083202504s) [seg_pos={0 0}, upload=506381]: Begin upload segment... Dec 29 17:07:48.212181948 (577h38m48.083264993s) [seg_pos={0 0}, upload=506381]: Done begin upload segment. Dec 29 17:07:48.212195984 (577h38m48.083279019s) [noise=true , address=145.79.227.48:4302 , piece_id=LEPF345NW7BXUKAUDWTKAQHHAI3QKVYDNVTAT34HC54NCMBTOKZA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.212269955 (577h38m48.083353s) [noise=true , address=45.91.116.11:4119 , piece_id=CUZDT474FH2IGZDUXHDIZWAOXSROIBG545FUB7UY4RPVRX7BHUOA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.212313067 (577h38m48.083396102s) [noise=true , address=45.91.116.7:4012 , piece_id=WCIMM4M7YW673RPTBQE5DE3AFOONG2ROO5ITJBN4CDDG6K2DJAWQ, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.212609512 (577h38m48.083692577s) [noise=true , address=145.79.227.22:4509 , piece_id=LIOHMVEENLXHMCWFRGLHCQPKMW77WMLKN2DVSWO5VMJON2HJS6KQ, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.213089807 (577h38m48.084172852s) [noise=true , address=145.79.227.26:4654 , piece_id=75D3S3OSP3SB7E2YUQIO2IDZED3Z6E5SLXXXAK6C5IPJX6EYNC2A, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.213457256 (577h38m48.084540301s) [noise=true , address=45.91.116.12:4227 , piece_id=JF5LSTOEOVBFRUSYS2S7ATQSZ3XN4OQIM45OA3UWLIW2MSZZF6EA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.213882687 (577h38m48.084965722s) [noise=true , address=145.79.227.7:4293 , piece_id=43K645GSPMCBXCWNYEBIXVJJGBMR3UXCWSURM4RVN77OBNEZLHZQ, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.214281506 (577h38m48.085364541s) [noise=true , address=145.79.227.8:4329 , piece_id=NUF75NGV6YGIZJPSL633MOT3JD6TRJYD5J253IUC3NOB4DIUZQTA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.214718719 (577h38m48.085801754s) [noise=true , address=145.79.227.47:4267 , piece_id=5NARPVAC2XQQTVPNNI3VBFBUZAEHTAC2NSIDGRINAX4CT7QRR42Q, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.215164346 (577h38m48.086247391s) [noise=true , address=145.79.227.32:4836 , piece_id=Q4YPYNZ57JLMAIIBPTGY4BZB223YZSRPISA3YQGKAWBBMC3ICF7Q, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.215626838 (577h38m48.086709873s) [noise=true , address=145.79.227.29:4943 , piece_id=774PLDODJ7KXRYW2XLHZJDU7GMWPMJ4YW43YFMFUMJJAR2PXCJCQ, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.216229575 (577h38m48.08731262s) [noise=true , address=77.90.31.13:4626 , piece_id=GIMBYYCTGEG5RP2ERG7AYU4R5DJ4TQIAIFL6DW5LRCR2ZL5NSZHA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.217852776 (577h38m48.088935821s) [noise=true , address=77.90.31.12:4590 , piece_id=4LARD25LAT5QVY2MOUP5PUUZTRA5EV5XDZI65BVB73RKGBNXEDOA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.219360607 (577h38m48.090443642s) [noise=true , address=77.90.31.16:4736 , piece_id=OXEU3PGRU4S6KLKPP6UDDLW66MNIC5IZRKWYCAWKVYEOFMXCDHVQ, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.222860210 (577h38m48.093943246s) [noise=true , address=145.79.227.50:4374 , piece_id=XBJD4EVV55MF5TIGQ5VSWICACTMDG4QQD2HIRYXRX2TWAYE5J4WA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:48.224115943 (577h38m48.095198988s) [noise=true , address=145.79.227.11:4729 , piece_id=L4FVUPE5CVMJS6KZIMCFSYF4B267YNRBQQCPWNBF3PF3SOZDVTHA, seg_pos={0 0}, upload=506381]: Uploading piece... Dec 29 17:07:49.882991127 (577h38m49.754074162s) [upload=506381]: Next returned nil segment Dec 29 17:07:49.883020533 (577h38m49.754103568s) [upload=506381]: Waiting for error group managing segments... Dec 29 17:07:50.192047537 (577h38m50.063130572s) [noise=true , address=145.79.227.48:4302 , piece_id=LEPF345NW7BXUKAUDWTKAQHHAI3QKVYDNVTAT34HC54NCMBTOKZA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.196257343 (577h38m50.067340378s) [noise=true , address=145.79.227.22:4509 , piece_id=LIOHMVEENLXHMCWFRGLHCQPKMW77WMLKN2DVSWO5VMJON2HJS6KQ, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.206488776 (577h38m50.077571811s) [noise=true , address=145.79.227.8:4329 , piece_id=NUF75NGV6YGIZJPSL633MOT3JD6TRJYD5J253IUC3NOB4DIUZQTA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.222883323 (577h38m50.093966358s) [noise=true , address=145.79.227.47:4267 , piece_id=5NARPVAC2XQQTVPNNI3VBFBUZAEHTAC2NSIDGRINAX4CT7QRR42Q, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.227113558 (577h38m50.098196593s) [noise=true , address=145.79.227.26:4654 , piece_id=75D3S3OSP3SB7E2YUQIO2IDZED3Z6E5SLXXXAK6C5IPJX6EYNC2A, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.229407336 (577h38m50.100490371s) [noise=true , address=145.79.227.50:4374 , piece_id=XBJD4EVV55MF5TIGQ5VSWICACTMDG4QQD2HIRYXRX2TWAYE5J4WA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.230106537 (577h38m50.101189572s) [noise=true , address=145.79.227.32:4836 , piece_id=Q4YPYNZ57JLMAIIBPTGY4BZB223YZSRPISA3YQGKAWBBMC3ICF7Q, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.232057552 (577h38m50.103140587s) [noise=true , address=145.79.227.11:4729 , piece_id=L4FVUPE5CVMJS6KZIMCFSYF4B267YNRBQQCPWNBF3PF3SOZDVTHA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.235549341 (577h38m50.106632376s) [noise=true , address=145.79.227.7:4293 , piece_id=43K645GSPMCBXCWNYEBIXVJJGBMR3UXCWSURM4RVN77OBNEZLHZQ, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.385884772 (577h38m50.256967817s) [noise=true , address=77.90.31.12:4590 , piece_id=4LARD25LAT5QVY2MOUP5PUUZTRA5EV5XDZI65BVB73RKGBNXEDOA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.455632382 (577h38m50.326715417s) [noise=true , address=45.91.116.7:4012 , piece_id=WCIMM4M7YW673RPTBQE5DE3AFOONG2ROO5ITJBN4CDDG6K2DJAWQ, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.461882363 (577h38m50.332965398s) [noise=true , address=45.91.116.12:4227 , piece_id=JF5LSTOEOVBFRUSYS2S7ATQSZ3XN4OQIM45OA3UWLIW2MSZZF6EA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.467516471 (577h38m50.338599506s) [noise=true , address=77.90.31.16:4736 , piece_id=OXEU3PGRU4S6KLKPP6UDDLW66MNIC5IZRKWYCAWKVYEOFMXCDHVQ, seg_pos={0 0}, upload=506381]: Done uploading piece. err: <nil> Dec 29 17:07:50.467533483 (577h38m50.338616518s) [seg_pos={0 0}, upload=506381]: Segment reached optimal threshold of 13 pieces. Dec 29 17:07:50.467723796 (577h38m50.338806831s) [noise=true , address=45.91.116.11:4119 , piece_id=CUZDT474FH2IGZDUXHDIZWAOXSROIBG545FUB7UY4RPVRX7BHUOA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:52330->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:52330->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52330->45.91.116.11:4119: use of closed network connection Dec 29 17:07:50.467729747 (577h38m50.338812782s) [noise=true , address=45.91.116.11:4119 , piece_id=CUZDT474FH2IGZDUXHDIZWAOXSROIBG545FUB7UY4RPVRX7BHUOA, seg_pos={0 0}, upload=506381]: Long tail cancellation due to optimal threshold Dec 29 17:07:50.467736880 (577h38m50.338819915s) [noise=true , address=77.90.31.13:4626 , piece_id=GIMBYYCTGEG5RP2ERG7AYU4R5DJ4TQIAIFL6DW5LRCR2ZL5NSZHA, seg_pos={0 0}, upload=506381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:35616->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:35616->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35616->77.90.31.13:4626: use of closed network connection Dec 29 17:07:50.467744596 (577h38m50.338827631s) [noise=true , address=77.90.31.13:4626 , piece_id=GIMBYYCTGEG5RP2ERG7AYU4R5DJ4TQIAIFL6DW5LRCR2ZL5NSZHA, seg_pos={0 0}, upload=506381]: Long tail cancellation due to optimal threshold Dec 29 17:07:50.467929568 (577h38m50.339012613s) [noise=true , address=145.79.227.29:4943 , piece_id=774PLDODJ7KXRYW2XLHZJDU7GMWPMJ4YW43YFMFUMJJAR2PXCJCQ, seg_pos={0 0}, upload=506381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 17:07:50.467938134 (577h38m50.339021169s) [noise=true , address=145.79.227.29:4943 , piece_id=774PLDODJ7KXRYW2XLHZJDU7GMWPMJ4YW43YFMFUMJJAR2PXCJCQ, seg_pos={0 0}, upload=506381]: Long tail cancellation due to optimal threshold Dec 29 17:07:50.467979152 (577h38m50.339062187s) [seg_pos={0 0}, upload=506381]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:07:50.468085384 (577h38m50.339168429s) [upload=506381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:07:50.468091064 (577h38m50.3391741s) [upload=506381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:07:50.762286695 (577h38m50.633369741s) [upload=506381]: Done uploading segments... Dec 29 17:07:50.762306774 (577h38m50.633389809s) [upload=506381]: Upload finished. err: <nil> Duration: 30.003299203s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX.] duration: 4.873267194s trace: https://jaeger.dev.storj.io/trace/7023c7d617723c7 FAIL (2025-12-29 17:07:48 UTC) orbiter warning: context deadline exceeded Duration: 4.873292534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.......X......................X........X...] duration: 43.111372097s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 43.111398889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [.X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X...] duration: 5.855200882s trace: https://jaeger.dev.storj.io/trace/7bcbfaf66752c341 FAIL (2025-12-29 17:07:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 29 17:05:24.804015542 (577h36m24.675098577s) [upload=506343]: Starting upload Dec 29 17:05:24.804068392 (577h36m24.675151437s) [upload=506343]: Done starting upload Dec 29 17:05:24.804161159 (577h36m24.675244204s) [upload=506343]: Uploading segments... Dec 29 17:05:24.804261791 (577h36m24.675344836s) [upload=506343]: Got next segment. Inline: false Dec 29 17:05:24.804315413 (577h36m24.675398448s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:05:24.804320442 (577h36m24.675403477s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:05:25.052373622 (577h36m24.923456668s) [seg_pos={0 0}, upload=506343]: Begin upload segment... Dec 29 17:05:25.052443375 (577h36m24.92352642s) [seg_pos={0 0}, upload=506343]: Done begin upload segment. Dec 29 17:05:25.052479073 (577h36m24.923562108s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.052552052 (577h36m24.923635087s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053171712 (577h36m24.924254757s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053577785 (577h36m24.92466082s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054019386 (577h36m24.925102431s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054452911 (577h36m24.925535956s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054466107 (577h36m24.925549152s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054912176 (577h36m24.925995211s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055225443 (577h36m24.926308478s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055520815 (577h36m24.92660385s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055798093 (577h36m24.926881128s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056085480 (577h36m24.927168525s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056434895 (577h36m24.92751793s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057596097 (577h36m24.928679132s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057985087 (577h36m24.929068133s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.059907899 (577h36m24.930990934s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:26.748197493 (577h36m26.619280528s) [upload=506343]: Next returned nil segment Dec 29 17:05:26.748225797 (577h36m26.619308843s) [upload=506343]: Waiting for error group managing segments... Dec 29 17:05:27.034151182 (577h36m26.905234217s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.038703930 (577h36m26.909786976s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.039376271 (577h36m26.910459306s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.047537111 (577h36m26.918620156s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.051494566 (577h36m26.922577601s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.056592213 (577h36m26.927675258s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.059394559 (577h36m26.930477604s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.072153914 (577h36m26.943236959s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.089761733 (577h36m26.960844778s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.286062809 (577h36m27.157145844s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.364053305 (577h36m27.23513634s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.920804592 (577h36m27.791887627s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938455672 (577h36m27.809538707s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938480881 (577h36m27.809563926s) [seg_pos={0 0}, upload=506343]: Segment reached optimal threshold of 13 pieces. Dec 29 17:05:27.938778717 (577h36m27.809861762s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection Dec 29 17:05:27.938787954 (577h36m27.809870989s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.938885230 (577h36m27.809968275s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.938905589 (577h36m27.809988634s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939224957 (577h36m27.810308002s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.939235487 (577h36m27.810318532s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939259653 (577h36m27.810342698s) [seg_pos={0 0}, upload=506343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:05:27.939374942 (577h36m27.810457987s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:05:27.939391985 (577h36m27.81047502s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:05:28.238480943 (577h36m28.109563988s) [upload=506343]: Done uploading segments... Dec 29 17:05:28.238509217 (577h36m28.109592262s) [upload=506343]: Upload finished. err: <nil> Duration: 5.855233634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X..] duration: 6.423447727s trace: https://jaeger.dev.storj.io/trace/7bcbfaf66752c341 FAIL (2025-12-29 17:06:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 29 17:05:24.804015542 (577h36m24.675098577s) [upload=506343]: Starting upload Dec 29 17:05:24.804068392 (577h36m24.675151437s) [upload=506343]: Done starting upload Dec 29 17:05:24.804161159 (577h36m24.675244204s) [upload=506343]: Uploading segments... Dec 29 17:05:24.804261791 (577h36m24.675344836s) [upload=506343]: Got next segment. Inline: false Dec 29 17:05:24.804315413 (577h36m24.675398448s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:05:24.804320442 (577h36m24.675403477s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:05:25.052373622 (577h36m24.923456668s) [seg_pos={0 0}, upload=506343]: Begin upload segment... Dec 29 17:05:25.052443375 (577h36m24.92352642s) [seg_pos={0 0}, upload=506343]: Done begin upload segment. Dec 29 17:05:25.052479073 (577h36m24.923562108s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.052552052 (577h36m24.923635087s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053171712 (577h36m24.924254757s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053577785 (577h36m24.92466082s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054019386 (577h36m24.925102431s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054452911 (577h36m24.925535956s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054466107 (577h36m24.925549152s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054912176 (577h36m24.925995211s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055225443 (577h36m24.926308478s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055520815 (577h36m24.92660385s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055798093 (577h36m24.926881128s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056085480 (577h36m24.927168525s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056434895 (577h36m24.92751793s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057596097 (577h36m24.928679132s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057985087 (577h36m24.929068133s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.059907899 (577h36m24.930990934s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:26.748197493 (577h36m26.619280528s) [upload=506343]: Next returned nil segment Dec 29 17:05:26.748225797 (577h36m26.619308843s) [upload=506343]: Waiting for error group managing segments... Dec 29 17:05:27.034151182 (577h36m26.905234217s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.038703930 (577h36m26.909786976s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.039376271 (577h36m26.910459306s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.047537111 (577h36m26.918620156s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.051494566 (577h36m26.922577601s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.056592213 (577h36m26.927675258s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.059394559 (577h36m26.930477604s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.072153914 (577h36m26.943236959s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.089761733 (577h36m26.960844778s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.286062809 (577h36m27.157145844s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.364053305 (577h36m27.23513634s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.920804592 (577h36m27.791887627s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938455672 (577h36m27.809538707s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938480881 (577h36m27.809563926s) [seg_pos={0 0}, upload=506343]: Segment reached optimal threshold of 13 pieces. Dec 29 17:05:27.938778717 (577h36m27.809861762s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection Dec 29 17:05:27.938787954 (577h36m27.809870989s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.938885230 (577h36m27.809968275s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.938905589 (577h36m27.809988634s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939224957 (577h36m27.810308002s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.939235487 (577h36m27.810318532s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939259653 (577h36m27.810342698s) [seg_pos={0 0}, upload=506343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:05:27.939374942 (577h36m27.810457987s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:05:27.939391985 (577h36m27.81047502s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:05:28.238480943 (577h36m28.109563988s) [upload=506343]: Done uploading segments... Dec 29 17:05:28.238509217 (577h36m28.109592262s) [upload=506343]: Upload finished. err: <nil> Duration: 6.423467625s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXXX] duration: 5.000722948s trace: https://jaeger.dev.storj.io/trace/7023c7d617723c7 FAIL (2025-12-29 17:06:38 UTC) orbiter warning: context deadline exceeded Duration: 5.000742525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [...X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X.] duration: 6.119299705s trace: https://jaeger.dev.storj.io/trace/7bcbfaf66752c341 FAIL (2025-12-29 17:06:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 29 17:05:24.804015542 (577h36m24.675098577s) [upload=506343]: Starting upload Dec 29 17:05:24.804068392 (577h36m24.675151437s) [upload=506343]: Done starting upload Dec 29 17:05:24.804161159 (577h36m24.675244204s) [upload=506343]: Uploading segments... Dec 29 17:05:24.804261791 (577h36m24.675344836s) [upload=506343]: Got next segment. Inline: false Dec 29 17:05:24.804315413 (577h36m24.675398448s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:05:24.804320442 (577h36m24.675403477s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:05:25.052373622 (577h36m24.923456668s) [seg_pos={0 0}, upload=506343]: Begin upload segment... Dec 29 17:05:25.052443375 (577h36m24.92352642s) [seg_pos={0 0}, upload=506343]: Done begin upload segment. Dec 29 17:05:25.052479073 (577h36m24.923562108s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.052552052 (577h36m24.923635087s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053171712 (577h36m24.924254757s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053577785 (577h36m24.92466082s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054019386 (577h36m24.925102431s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054452911 (577h36m24.925535956s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054466107 (577h36m24.925549152s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054912176 (577h36m24.925995211s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055225443 (577h36m24.926308478s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055520815 (577h36m24.92660385s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055798093 (577h36m24.926881128s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056085480 (577h36m24.927168525s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056434895 (577h36m24.92751793s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057596097 (577h36m24.928679132s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057985087 (577h36m24.929068133s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.059907899 (577h36m24.930990934s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:26.748197493 (577h36m26.619280528s) [upload=506343]: Next returned nil segment Dec 29 17:05:26.748225797 (577h36m26.619308843s) [upload=506343]: Waiting for error group managing segments... Dec 29 17:05:27.034151182 (577h36m26.905234217s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.038703930 (577h36m26.909786976s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.039376271 (577h36m26.910459306s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.047537111 (577h36m26.918620156s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.051494566 (577h36m26.922577601s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.056592213 (577h36m26.927675258s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.059394559 (577h36m26.930477604s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.072153914 (577h36m26.943236959s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.089761733 (577h36m26.960844778s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.286062809 (577h36m27.157145844s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.364053305 (577h36m27.23513634s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.920804592 (577h36m27.791887627s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938455672 (577h36m27.809538707s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938480881 (577h36m27.809563926s) [seg_pos={0 0}, upload=506343]: Segment reached optimal threshold of 13 pieces. Dec 29 17:05:27.938778717 (577h36m27.809861762s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection Dec 29 17:05:27.938787954 (577h36m27.809870989s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.938885230 (577h36m27.809968275s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.938905589 (577h36m27.809988634s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939224957 (577h36m27.810308002s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.939235487 (577h36m27.810318532s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939259653 (577h36m27.810342698s) [seg_pos={0 0}, upload=506343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:05:27.939374942 (577h36m27.810457987s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:05:27.939391985 (577h36m27.81047502s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:05:28.238480943 (577h36m28.109563988s) [upload=506343]: Done uploading segments... Dec 29 17:05:28.238509217 (577h36m28.109592262s) [upload=506343]: Upload finished. err: <nil> Duration: 6.11932404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.......X......................X........X..] duration: 15.456114365s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:06:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 15.45616886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.X] duration: 30.002868286s trace: https://jaeger.dev.storj.io/trace/7bcbfaf66752c341 FAIL (2025-12-29 17:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 29 17:05:24.804015542 (577h36m24.675098577s) [upload=506343]: Starting upload Dec 29 17:05:24.804068392 (577h36m24.675151437s) [upload=506343]: Done starting upload Dec 29 17:05:24.804161159 (577h36m24.675244204s) [upload=506343]: Uploading segments... Dec 29 17:05:24.804261791 (577h36m24.675344836s) [upload=506343]: Got next segment. Inline: false Dec 29 17:05:24.804315413 (577h36m24.675398448s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:05:24.804320442 (577h36m24.675403477s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:05:25.052373622 (577h36m24.923456668s) [seg_pos={0 0}, upload=506343]: Begin upload segment... Dec 29 17:05:25.052443375 (577h36m24.92352642s) [seg_pos={0 0}, upload=506343]: Done begin upload segment. Dec 29 17:05:25.052479073 (577h36m24.923562108s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.052552052 (577h36m24.923635087s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053171712 (577h36m24.924254757s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.053577785 (577h36m24.92466082s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054019386 (577h36m24.925102431s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054452911 (577h36m24.925535956s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054466107 (577h36m24.925549152s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.054912176 (577h36m24.925995211s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055225443 (577h36m24.926308478s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055520815 (577h36m24.92660385s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.055798093 (577h36m24.926881128s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056085480 (577h36m24.927168525s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.056434895 (577h36m24.92751793s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057596097 (577h36m24.928679132s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.057985087 (577h36m24.929068133s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:25.059907899 (577h36m24.930990934s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Uploading piece... Dec 29 17:05:26.748197493 (577h36m26.619280528s) [upload=506343]: Next returned nil segment Dec 29 17:05:26.748225797 (577h36m26.619308843s) [upload=506343]: Waiting for error group managing segments... Dec 29 17:05:27.034151182 (577h36m26.905234217s) [noise=true , address=145.79.227.50:4374 , piece_id=F55RXVFHMNKQAUZLLYDJZFWPKV7DH45SM7T5U5KKQDPO6LSMRC3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.038703930 (577h36m26.909786976s) [noise=true , address=145.79.227.8:4329 , piece_id=FELXKJBXC24Z5F4Q52YQ2SLXQ464RPHGXMBUOO457E46XNELMSTQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.039376271 (577h36m26.910459306s) [noise=true , address=145.79.227.41:4087 , piece_id=YCGP6M55H4WJVS7IZO5LY2W7KMHCHXFKF24UNGDX2YFQCHSZJOQA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.047537111 (577h36m26.918620156s) [noise=true , address=145.79.227.13:4804 , piece_id=IMT7UJ7VC5Z5JZTABBWKSDR6Y6BDBVC45HQE7MV2VZ76IOQJ5WTA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.051494566 (577h36m26.922577601s) [noise=true , address=145.79.227.24:4582 , piece_id=4TOHHWH3YTKYZHR3HGQUWKWG4JK2KYITZUUFQODL7COGGSZM6HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.056592213 (577h36m26.927675258s) [noise=true , address=145.79.227.36:4519 , piece_id=KP44XF3SU5B7T3ZOC62DEQBL5P2KRQMV75YLVYOCJK6GHSKEJTEQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.059394559 (577h36m26.930477604s) [noise=true , address=145.79.227.22:4509 , piece_id=4VFHWSEP5AXA526IFABARJTJHO2V3OUST2Q7R25DDMTHKYIIIQPA, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.072153914 (577h36m26.943236959s) [noise=true , address=145.79.227.38:4979 , piece_id=QS7TRTYKTMFQPCSJWGJABQNMU2CG2ZAONXQM3GVRVW35JIEG2T4Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.089761733 (577h36m26.960844778s) [noise=true , address=145.79.227.52:4447 , piece_id=SCKJMRJFVLB2RFRFPICRYZZB66EFTITBLNYYICZDA3VFYWC3PX3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.286062809 (577h36m27.157145844s) [noise=true , address=45.91.116.13:4154 , piece_id=7MUS2AYGVUPCXCKLKJV3MW4JVPGCHE5AZPBEX5IYUCPWNWT72HJQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.364053305 (577h36m27.23513634s) [noise=true , address=45.91.116.14:4191 , piece_id=KDCO7CYWHZJY2WAASFZCOE4BMVYIIK57CCTE2SS2ZGSUQFZWA7MQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.920804592 (577h36m27.791887627s) [noise=true , address=77.90.31.10:4520 , piece_id=CX4C7MIKU2ND6LBHTBO2XBSJJX2CV6SRVLCGCOUR4FOFBYEIB7VQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938455672 (577h36m27.809538707s) [noise=true , address=45.91.116.6:4974 , piece_id=6CVBLKJNBGS5H4CBFUTE7G3PRJFE7TVTUMQQ6RYJXPKQCFLBDU3Q, seg_pos={0 0}, upload=506343]: Done uploading piece. err: <nil> Dec 29 17:05:27.938480881 (577h36m27.809563926s) [seg_pos={0 0}, upload=506343]: Segment reached optimal threshold of 13 pieces. Dec 29 17:05:27.938778717 (577h36m27.809861762s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47056->45.91.116.9:4263: use of closed network connection Dec 29 17:05:27.938787954 (577h36m27.809870989s) [noise=true , address=45.91.116.9:4263 , piece_id=FWYT4PB7RXNDLLRIKI3QBXK4J7JI4SLFWOM3GYPXPF4XIX2FRFAQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.938885230 (577h36m27.809968275s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.938905589 (577h36m27.809988634s) [noise=true , address=45.91.116.5:4297 , piece_id=P7BCE24KHI25TVAPKFLI36EWBOTDKPF6V2PO5ELGQOIKUC6MN6RQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939224957 (577h36m27.810308002s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 17:05:27.939235487 (577h36m27.810318532s) [noise=true , address=45.91.116.7:4011 , piece_id=ZQ4KL3CUJDH3OFP3PPAEFOKX5VMIBPOAUJ7O4IURB27G34GFVZVQ, seg_pos={0 0}, upload=506343]: Long tail cancellation due to optimal threshold Dec 29 17:05:27.939259653 (577h36m27.810342698s) [seg_pos={0 0}, upload=506343]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:05:27.939374942 (577h36m27.810457987s) [upload=506343]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:05:27.939391985 (577h36m27.81047502s) [upload=506343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:05:28.238480943 (577h36m28.109563988s) [upload=506343]: Done uploading segments... Dec 29 17:05:28.238509217 (577h36m28.109592262s) [upload=506343]: Upload finished. err: <nil> Duration: 30.002884577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [X.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXXX] duration: 5.001822831s trace: https://jaeger.dev.storj.io/trace/2a3f3081a7aba8f9 FAIL (2025-12-29 17:05:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001888105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.......X......................X........X.] duration: 35.285153013s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:05:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 35.285169996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.......X......................X........X] duration: 1m0.002958248s trace: https://jaeger.dev.storj.io/trace/7c7783590549082d FAIL (2025-12-29 17:04:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3:56.553681039 (577h34m56.424764074s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554319576 (577h34m56.425402621s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554351876 (577h34m56.425434911s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554712393 (577h34m56.425795438s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.554811532 (577h34m56.425894577s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555082047 (577h34m56.426165092s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555150717 (577h34m56.426233762s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555512086 (577h34m56.426595121s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.555944439 (577h34m56.427027484s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556213051 (577h34m56.427296086s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556550844 (577h34m56.427633879s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.556985291 (577h34m56.428068326s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557124616 (577h34m56.428207651s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557511523 (577h34m56.428594558s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.557752342 (577h34m56.428835377s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558130201 (577h34m56.429213236s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:56.558444870 (577h34m56.429527915s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Uploading piece... Dec 29 17:03:57.916384610 (577h34m57.787467655s) [upload=506323]: Next returned nil segment Dec 29 17:03:57.916396592 (577h34m57.787479638s) [upload=506323]: Waiting for error group managing segments... Dec 29 17:03:58.408503477 (577h34m58.279586512s) [noise=true , address=145.79.227.12:4790 , piece_id=JHB4BJENOJTAHVENZHYB7QEYWPBY6EZ6R53FCUF654PM43EMNIQQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.429561024 (577h34m58.300644079s) [noise=true , address=145.79.227.25:4650 , piece_id=ZJBIU3NOEYQGG7VVMQJSTQSE4XV4KQWHKOPJ5V5DB4NQFKE3SARA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.441695730 (577h34m58.312778775s) [noise=true , address=64.49.13.170:4720 , piece_id=CNHWT7KBN3766UPGKQZKLR557LQR6IF5F56IYRRVOUJ64LUWNRFA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.451903457 (577h34m58.322986492s) [noise=true , address=145.79.227.27:4703 , piece_id=B3AICSGQNJNARHWNAIZBBTMSN3BZEFZED4EDUQB2F27EM7D7M67Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.577693920 (577h34m58.448776965s) [noise=true , address=64.49.13.140:4516 , piece_id=LNMUQLYWFRMULMVCZRTYBJVFOR2JFMK4ELE2O3QHCDXHAOBA4Y7Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.677877535 (577h34m58.54896057s) [noise=true , address=45.91.116.14:4207 , piece_id=URY32AWWSUD3RRY4WXD3JHN4QENTRJEIILBZDGFA6DLT74T6WYZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.746257871 (577h34m58.617340916s) [noise=true , address=77.90.31.12:4599 , piece_id=4HEHDOS6I4XLAAZQDYEWALX5GFZUJYR5HORM5F5O5YGVUFPBGSVQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:58.751112405 (577h34m58.62219545s) [noise=true , address=77.90.31.21:4884 , piece_id=SWFBNVUYKPEGZMH7JVLQ7YYE4E2PFJLADIJCUHEPL7P5IWXVSVKQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.220597253 (577h34m59.091680298s) [noise=true , address=45.91.116.8:4055 , piece_id=6OUESLPI2XGP2ZALSYDLHRR3GP66J4SC3LAQMZJWWFS7M7LKUNZQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.316134560 (577h34m59.187217605s) [noise=true , address=145.79.227.29:4970 , piece_id=TZEQRN6KLUFORPSQNSXJFNXEVUGF55DNGMFHTJYHCPNYJ76GTIYA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.351755158 (577h34m59.222838213s) [noise=true , address=145.79.227.29:4949 , piece_id=7CUNAFLEBUB5VTVVWBXBVH5536RCI3A4WQ56QHHPW3PCCW6RBVFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.366579245 (577h34m59.23766229s) [noise=true , address=145.79.227.44:4218 , piece_id=G3MY2HYBD6FOI3DVZKYR7HIVSMED3S6CFNPY2PW6BQ2KHWVWFBWA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.417114402 (577h34m59.288197437s) [noise=true , address=145.79.227.49:4370 , piece_id=M3EANCE7MPYMUXE32SMIH5AQK6ZCCTWC3NNJSN47C57VHR3VSILQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447459413 (577h34m59.318542458s) [noise=true , address=145.79.227.34:4937 , piece_id=XY5D7LNXFVISP7BD7PWF7J6AIHJX2PAX6C6XHMXHBN25BR4YI2TQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: <nil> Dec 29 17:03:59.447484400 (577h34m59.318567445s) [seg_pos={0 0}, upload=506323]: Segment reached optimal threshold of 14 pieces. Dec 29 17:03:59.447673741 (577h34m59.318756776s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447681706 (577h34m59.318764741s) [noise=true , address=145.79.227.33:4901 , piece_id=RND3O75ATKV5VBUZMQQF3WGGYBOUDA5FWQ7YKQYTEJEVZ7DPCJFQ, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447796995 (577h34m59.31888004s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447802827 (577h34m59.318885862s) [noise=true , address=145.79.227.40:4068 , piece_id=WGM2QQ4GM2FEVE3URP5UP3MRN3EEMCALESBZU3CM4EO6QJUSFM4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.447903778 (577h34m59.318986823s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.447908989 (577h34m59.318992024s) [noise=true , address=145.79.227.43:4169 , piece_id=GUMKVAAD6B35Y7X3TOFJC75QP6WEC6LD6XANKZEIZKEPPM574U4Q, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448113428 (577h34m59.319196493s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:59.448133356 (577h34m59.319216401s) [noise=true , address=193.149.164.143:4700 , piece_id=OW3G5CDC2EQN3QKL3Y63XQIDGH2KSHWULMVCHPPWQPKJ36SYU6VA, seg_pos={0 0}, upload=506323]: Long tail cancellation due to optimal threshold Dec 29 17:03:59.448192389 (577h34m59.319275434s) [seg_pos={0 0}, upload=506323]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 17:03:59.448322325 (577h34m59.31940537s) [upload=506323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:59.448331844 (577h34m59.319414939s) [upload=506323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:59.745988226 (577h34m59.617071281s) [upload=506323]: Done uploading segments... Dec 29 17:03:59.746011389 (577h34m59.617094434s) [upload=506323]: Upload finished. err: <nil> Duration: 1m0.002969939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXXX] duration: 5.00148936s trace: https://jaeger.dev.storj.io/trace/4adb6f0187bfbb12 FAIL (2025-12-29 17:04:47 UTC) orbiter warning: context deadline exceeded Duration: 5.001511641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [X....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X.] duration: 6.107635674s trace: https://jaeger.dev.storj.io/trace/4e4d2dc6fb9a0796 FAIL (2025-12-29 17:04:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: G3IGYDJTAP2MFQRAF7KPEAZKQ3STZC5SH4TSEOUINSQWV6PZGXYQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 7ED7YQNKX5LAJYBNG6JOQMAEYLQKVVWLXD7VS7YQCNW6DQA6S6PA) context deadline exceeded ======LOGS====== Dec 29 17:03:22.228347660 (577h34m22.099430695s) [upload=506314]: Starting upload Dec 29 17:03:22.228540447 (577h34m22.099623493s) [upload=506314]: Done starting upload Dec 29 17:03:22.228624948 (577h34m22.099707983s) [upload=506314]: Uploading segments... Dec 29 17:03:22.228960978 (577h34m22.100044013s) [upload=506314]: Got next segment. Inline: false Dec 29 17:03:22.229022955 (577h34m22.10010599s) [upload=506314]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:03:22.229029559 (577h34m22.100112604s) [upload=506314]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:03:22.484132112 (577h34m22.355215147s) [seg_pos={0 0}, upload=506314]: Begin upload segment... Dec 29 17:03:22.484190152 (577h34m22.355273197s) [seg_pos={0 0}, upload=506314]: Done begin upload segment. Dec 29 17:03:22.484342092 (577h34m22.355425127s) [noise=true , address=77.90.31.16:4736 , piece_id=G3IGYDJTAP2MFQRAF7KPEAZKQ3STZC5SH4TSEOUINSQWV6PZGXYQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.484433856 (577h34m22.355516891s) [noise=true , address=145.79.227.22:4509 , piece_id=HL5LWGPZDDTN4DBLQSLV7T6O4E5RRBIWVWY2QRNPVWD7SZ43GNKA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.484924160 (577h34m22.356007195s) [noise=true , address=145.79.227.46:4603 , piece_id=BJXF336JTIITB3BQ2TXGZ2LHOKY6NCUDDWAK3QBSK7ULBM6Z5NCQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.485581090 (577h34m22.356664135s) [noise=true , address=145.79.227.52:4445 , piece_id=PYONF72PG7AXJ4OS47W52DYJVR75JJPZZJ76GIWWWQ7S5P2NAO6A, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.486176425 (577h34m22.35725947s) [noise=true , address=45.91.116.6:4976 , piece_id=UZK43N75RSDHU73LMFFOHTVED65MRGY72E5DLBK2OGAMALU4LKTQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.486726922 (577h34m22.357809977s) [noise=true , address=145.79.227.41:4085 , piece_id=4MZQOJTPBOCLQZCSTZUXTK33KQI6DNBQT37OQYGPYJC46VFNBAXQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.487339509 (577h34m22.358422554s) [noise=true , address=145.79.227.26:4655 , piece_id=SBF2TRCSJT42IXY6AZEIAURWHGXDSLUQUL3OHLCUPNM4OCJ4VJYQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.488882126 (577h34m22.359965161s) [noise=true , address=145.79.227.4:4185 , piece_id=ANNTUFXJQ5HQ6AQSUHE7VMWOITJBNCWBLBS6QGOBPXVOGH5MJXZA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.490522640 (577h34m22.361605695s) [noise=true , address=77.90.31.12:4590 , piece_id=KZURRXFBRWWKZA5FEDCDTMQJ6IL3K7JCMQQQF7AC2YJKHG7RHEVA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.492266931 (577h34m22.363349976s) [noise=true , address=77.90.31.18:4805 , piece_id=2DTKHJDUXOHD4NKINQCMKF6M3WPUCYJTMZLPYH6XNYWNWOS6KZXQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.493163338 (577h34m22.364246383s) [noise=true , address=145.79.227.15:4874 , piece_id=WD7RZTQ2ZTMKPM326LOIOQFXOYP4JOAGW3A4BMKZVQZOY7SC77WQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.494158213 (577h34m22.365241258s) [noise=true , address=145.79.227.2:4115 , piece_id=PP6JMNSYXQEKMT2TPF2AOP5G66TMSWZOJISUQ5OPVLRU7M4EGEYQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.494498481 (577h34m22.365581516s) [noise=true , address=145.79.227.16:4909 , piece_id=CZHCV4FBRISEMWNAUKSPDI74TPU46SHJACRR5VK5VDXEXS3TJZOA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.495564350 (577h34m22.366647385s) [noise=true , address=77.90.31.17:4770 , piece_id=7ED7YQNKX5LAJYBNG6JOQMAEYLQKVVWLXD7VS7YQCNW6DQA6S6PA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.496582128 (577h34m22.367665183s) [noise=true , address=45.91.116.7:4011 , piece_id=QVHQ4KCWCKFMAU6ONHTEFVPHEUJ3Q6T7QAXBES5OFIPW6ET2CXSQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.496951642 (577h34m22.368034687s) [noise=true , address=145.79.227.11:4730 , piece_id=JAUTX2IUPJM2OQMFBF3L7S466BVVLDK6TR4HSJIR6PWCW3HLEPVA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:24.186274364 (577h34m24.057357409s) [upload=506314]: Next returned nil segment Dec 29 17:03:24.186289402 (577h34m24.057372437s) [upload=506314]: Waiting for error group managing segments... Dec 29 17:03:24.471711308 (577h34m24.342794343s) [noise=true , address=145.79.227.41:4085 , piece_id=4MZQOJTPBOCLQZCSTZUXTK33KQI6DNBQT37OQYGPYJC46VFNBAXQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.472161484 (577h34m24.343244529s) [noise=true , address=145.79.227.2:4115 , piece_id=PP6JMNSYXQEKMT2TPF2AOP5G66TMSWZOJISUQ5OPVLRU7M4EGEYQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.474565081 (577h34m24.345648126s) [noise=true , address=145.79.227.11:4730 , piece_id=JAUTX2IUPJM2OQMFBF3L7S466BVVLDK6TR4HSJIR6PWCW3HLEPVA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.487887840 (577h34m24.358970885s) [noise=true , address=145.79.227.46:4603 , piece_id=BJXF336JTIITB3BQ2TXGZ2LHOKY6NCUDDWAK3QBSK7ULBM6Z5NCQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.487978502 (577h34m24.359061537s) [noise=true , address=145.79.227.52:4445 , piece_id=PYONF72PG7AXJ4OS47W52DYJVR75JJPZZJ76GIWWWQ7S5P2NAO6A, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.488440272 (577h34m24.359523307s) [noise=true , address=145.79.227.22:4509 , piece_id=HL5LWGPZDDTN4DBLQSLV7T6O4E5RRBIWVWY2QRNPVWD7SZ43GNKA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.491685642 (577h34m24.362768707s) [noise=true , address=145.79.227.26:4655 , piece_id=SBF2TRCSJT42IXY6AZEIAURWHGXDSLUQUL3OHLCUPNM4OCJ4VJYQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.492812768 (577h34m24.363895813s) [noise=true , address=145.79.227.15:4874 , piece_id=WD7RZTQ2ZTMKPM326LOIOQFXOYP4JOAGW3A4BMKZVQZOY7SC77WQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.511734377 (577h34m24.382817422s) [noise=true , address=145.79.227.16:4909 , piece_id=CZHCV4FBRISEMWNAUKSPDI74TPU46SHJACRR5VK5VDXEXS3TJZOA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.589864588 (577h34m24.460947643s) [noise=true , address=145.79.227.4:4185 , piece_id=ANNTUFXJQ5HQ6AQSUHE7VMWOITJBNCWBLBS6QGOBPXVOGH5MJXZA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.074814243 (577h34m24.945897278s) [noise=true , address=77.90.31.16:4736 , piece_id=G3IGYDJTAP2MFQRAF7KPEAZKQ3STZC5SH4TSEOUINSQWV6PZGXYQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.137658167 (577h34m25.008741202s) [noise=true , address=45.91.116.6:4976 , piece_id=UZK43N75RSDHU73LMFFOHTVED65MRGY72E5DLBK2OGAMALU4LKTQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.333824026 (577h34m25.204907061s) [noise=true , address=77.90.31.17:4770 , piece_id=7ED7YQNKX5LAJYBNG6JOQMAEYLQKVVWLXD7VS7YQCNW6DQA6S6PA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.333849434 (577h34m25.20493248s) [seg_pos={0 0}, upload=506314]: Segment reached optimal threshold of 13 pieces. Dec 29 17:03:25.334062741 (577h34m25.205145776s) [noise=true , address=77.90.31.12:4590 , piece_id=KZURRXFBRWWKZA5FEDCDTMQJ6IL3K7JCMQQQF7AC2YJKHG7RHEVA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:25.334071246 (577h34m25.205154282s) [noise=true , address=77.90.31.12:4590 , piece_id=KZURRXFBRWWKZA5FEDCDTMQJ6IL3K7JCMQQQF7AC2YJKHG7RHEVA, seg_pos={0 0}, upload=506314]: Long tail cancellation due to optimal threshold Dec 29 17:03:25.334418959 (577h34m25.205502004s) [noise=true , address=45.91.116.7:4011 , piece_id=QVHQ4KCWCKFMAU6ONHTEFVPHEUJ3Q6T7QAXBES5OFIPW6ET2CXSQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:25.334428387 (577h34m25.205511422s) [noise=true , address=45.91.116.7:4011 , piece_id=QVHQ4KCWCKFMAU6ONHTEFVPHEUJ3Q6T7QAXBES5OFIPW6ET2CXSQ, seg_pos={0 0}, upload=506314]: Long tail cancellation due to optimal threshold Dec 29 17:03:25.334653125 (577h34m25.20573618s) [noise=true , address=77.90.31.18:4805 , piece_id=2DTKHJDUXOHD4NKINQCMKF6M3WPUCYJTMZLPYH6XNYWNWOS6KZXQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:25.334661541 (577h34m25.205744576s) [noise=true , address=77.90.31.18:4805 , piece_id=2DTKHJDUXOHD4NKINQCMKF6M3WPUCYJTMZLPYH6XNYWNWOS6KZXQ, seg_pos={0 0}, upload=506314]: Long tail cancellation due to optimal threshold Dec 29 17:03:25.334685336 (577h34m25.205768371s) [seg_pos={0 0}, upload=506314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:03:25.334775547 (577h34m25.205858592s) [upload=506314]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:25.334781880 (577h34m25.205864925s) [upload=506314]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:25.633187479 (577h34m25.504270514s) [upload=506314]: Done uploading segments... Dec 29 17:03:25.633208299 (577h34m25.504291354s) [upload=506314]: Upload finished. err: <nil> Duration: 6.10766522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X......................X........] duration: 27.685575517s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 17:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 27.685624451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..X] duration: 30.00294427s trace: https://jaeger.dev.storj.io/trace/4e4d2dc6fb9a0796 FAIL (2025-12-29 17:03:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: G3IGYDJTAP2MFQRAF7KPEAZKQ3STZC5SH4TSEOUINSQWV6PZGXYQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 7ED7YQNKX5LAJYBNG6JOQMAEYLQKVVWLXD7VS7YQCNW6DQA6S6PA) context deadline exceeded ======LOGS====== Dec 29 17:03:22.228347660 (577h34m22.099430695s) [upload=506314]: Starting upload Dec 29 17:03:22.228540447 (577h34m22.099623493s) [upload=506314]: Done starting upload Dec 29 17:03:22.228624948 (577h34m22.099707983s) [upload=506314]: Uploading segments... Dec 29 17:03:22.228960978 (577h34m22.100044013s) [upload=506314]: Got next segment. Inline: false Dec 29 17:03:22.229022955 (577h34m22.10010599s) [upload=506314]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:03:22.229029559 (577h34m22.100112604s) [upload=506314]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:03:22.484132112 (577h34m22.355215147s) [seg_pos={0 0}, upload=506314]: Begin upload segment... Dec 29 17:03:22.484190152 (577h34m22.355273197s) [seg_pos={0 0}, upload=506314]: Done begin upload segment. Dec 29 17:03:22.484342092 (577h34m22.355425127s) [noise=true , address=77.90.31.16:4736 , piece_id=G3IGYDJTAP2MFQRAF7KPEAZKQ3STZC5SH4TSEOUINSQWV6PZGXYQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.484433856 (577h34m22.355516891s) [noise=true , address=145.79.227.22:4509 , piece_id=HL5LWGPZDDTN4DBLQSLV7T6O4E5RRBIWVWY2QRNPVWD7SZ43GNKA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.484924160 (577h34m22.356007195s) [noise=true , address=145.79.227.46:4603 , piece_id=BJXF336JTIITB3BQ2TXGZ2LHOKY6NCUDDWAK3QBSK7ULBM6Z5NCQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.485581090 (577h34m22.356664135s) [noise=true , address=145.79.227.52:4445 , piece_id=PYONF72PG7AXJ4OS47W52DYJVR75JJPZZJ76GIWWWQ7S5P2NAO6A, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.486176425 (577h34m22.35725947s) [noise=true , address=45.91.116.6:4976 , piece_id=UZK43N75RSDHU73LMFFOHTVED65MRGY72E5DLBK2OGAMALU4LKTQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.486726922 (577h34m22.357809977s) [noise=true , address=145.79.227.41:4085 , piece_id=4MZQOJTPBOCLQZCSTZUXTK33KQI6DNBQT37OQYGPYJC46VFNBAXQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.487339509 (577h34m22.358422554s) [noise=true , address=145.79.227.26:4655 , piece_id=SBF2TRCSJT42IXY6AZEIAURWHGXDSLUQUL3OHLCUPNM4OCJ4VJYQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.488882126 (577h34m22.359965161s) [noise=true , address=145.79.227.4:4185 , piece_id=ANNTUFXJQ5HQ6AQSUHE7VMWOITJBNCWBLBS6QGOBPXVOGH5MJXZA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.490522640 (577h34m22.361605695s) [noise=true , address=77.90.31.12:4590 , piece_id=KZURRXFBRWWKZA5FEDCDTMQJ6IL3K7JCMQQQF7AC2YJKHG7RHEVA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.492266931 (577h34m22.363349976s) [noise=true , address=77.90.31.18:4805 , piece_id=2DTKHJDUXOHD4NKINQCMKF6M3WPUCYJTMZLPYH6XNYWNWOS6KZXQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.493163338 (577h34m22.364246383s) [noise=true , address=145.79.227.15:4874 , piece_id=WD7RZTQ2ZTMKPM326LOIOQFXOYP4JOAGW3A4BMKZVQZOY7SC77WQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.494158213 (577h34m22.365241258s) [noise=true , address=145.79.227.2:4115 , piece_id=PP6JMNSYXQEKMT2TPF2AOP5G66TMSWZOJISUQ5OPVLRU7M4EGEYQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.494498481 (577h34m22.365581516s) [noise=true , address=145.79.227.16:4909 , piece_id=CZHCV4FBRISEMWNAUKSPDI74TPU46SHJACRR5VK5VDXEXS3TJZOA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.495564350 (577h34m22.366647385s) [noise=true , address=77.90.31.17:4770 , piece_id=7ED7YQNKX5LAJYBNG6JOQMAEYLQKVVWLXD7VS7YQCNW6DQA6S6PA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.496582128 (577h34m22.367665183s) [noise=true , address=45.91.116.7:4011 , piece_id=QVHQ4KCWCKFMAU6ONHTEFVPHEUJ3Q6T7QAXBES5OFIPW6ET2CXSQ, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:22.496951642 (577h34m22.368034687s) [noise=true , address=145.79.227.11:4730 , piece_id=JAUTX2IUPJM2OQMFBF3L7S466BVVLDK6TR4HSJIR6PWCW3HLEPVA, seg_pos={0 0}, upload=506314]: Uploading piece... Dec 29 17:03:24.186274364 (577h34m24.057357409s) [upload=506314]: Next returned nil segment Dec 29 17:03:24.186289402 (577h34m24.057372437s) [upload=506314]: Waiting for error group managing segments... Dec 29 17:03:24.471711308 (577h34m24.342794343s) [noise=true , address=145.79.227.41:4085 , piece_id=4MZQOJTPBOCLQZCSTZUXTK33KQI6DNBQT37OQYGPYJC46VFNBAXQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.472161484 (577h34m24.343244529s) [noise=true , address=145.79.227.2:4115 , piece_id=PP6JMNSYXQEKMT2TPF2AOP5G66TMSWZOJISUQ5OPVLRU7M4EGEYQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.474565081 (577h34m24.345648126s) [noise=true , address=145.79.227.11:4730 , piece_id=JAUTX2IUPJM2OQMFBF3L7S466BVVLDK6TR4HSJIR6PWCW3HLEPVA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.487887840 (577h34m24.358970885s) [noise=true , address=145.79.227.46:4603 , piece_id=BJXF336JTIITB3BQ2TXGZ2LHOKY6NCUDDWAK3QBSK7ULBM6Z5NCQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.487978502 (577h34m24.359061537s) [noise=true , address=145.79.227.52:4445 , piece_id=PYONF72PG7AXJ4OS47W52DYJVR75JJPZZJ76GIWWWQ7S5P2NAO6A, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.488440272 (577h34m24.359523307s) [noise=true , address=145.79.227.22:4509 , piece_id=HL5LWGPZDDTN4DBLQSLV7T6O4E5RRBIWVWY2QRNPVWD7SZ43GNKA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.491685642 (577h34m24.362768707s) [noise=true , address=145.79.227.26:4655 , piece_id=SBF2TRCSJT42IXY6AZEIAURWHGXDSLUQUL3OHLCUPNM4OCJ4VJYQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.492812768 (577h34m24.363895813s) [noise=true , address=145.79.227.15:4874 , piece_id=WD7RZTQ2ZTMKPM326LOIOQFXOYP4JOAGW3A4BMKZVQZOY7SC77WQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.511734377 (577h34m24.382817422s) [noise=true , address=145.79.227.16:4909 , piece_id=CZHCV4FBRISEMWNAUKSPDI74TPU46SHJACRR5VK5VDXEXS3TJZOA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:24.589864588 (577h34m24.460947643s) [noise=true , address=145.79.227.4:4185 , piece_id=ANNTUFXJQ5HQ6AQSUHE7VMWOITJBNCWBLBS6QGOBPXVOGH5MJXZA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.074814243 (577h34m24.945897278s) [noise=true , address=77.90.31.16:4736 , piece_id=G3IGYDJTAP2MFQRAF7KPEAZKQ3STZC5SH4TSEOUINSQWV6PZGXYQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.137658167 (577h34m25.008741202s) [noise=true , address=45.91.116.6:4976 , piece_id=UZK43N75RSDHU73LMFFOHTVED65MRGY72E5DLBK2OGAMALU4LKTQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.333824026 (577h34m25.204907061s) [noise=true , address=77.90.31.17:4770 , piece_id=7ED7YQNKX5LAJYBNG6JOQMAEYLQKVVWLXD7VS7YQCNW6DQA6S6PA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: <nil> Dec 29 17:03:25.333849434 (577h34m25.20493248s) [seg_pos={0 0}, upload=506314]: Segment reached optimal threshold of 13 pieces. Dec 29 17:03:25.334062741 (577h34m25.205145776s) [noise=true , address=77.90.31.12:4590 , piece_id=KZURRXFBRWWKZA5FEDCDTMQJ6IL3K7JCMQQQF7AC2YJKHG7RHEVA, seg_pos={0 0}, upload=506314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:25.334071246 (577h34m25.205154282s) [noise=true , address=77.90.31.12:4590 , piece_id=KZURRXFBRWWKZA5FEDCDTMQJ6IL3K7JCMQQQF7AC2YJKHG7RHEVA, seg_pos={0 0}, upload=506314]: Long tail cancellation due to optimal threshold Dec 29 17:03:25.334418959 (577h34m25.205502004s) [noise=true , address=45.91.116.7:4011 , piece_id=QVHQ4KCWCKFMAU6ONHTEFVPHEUJ3Q6T7QAXBES5OFIPW6ET2CXSQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:25.334428387 (577h34m25.205511422s) [noise=true , address=45.91.116.7:4011 , piece_id=QVHQ4KCWCKFMAU6ONHTEFVPHEUJ3Q6T7QAXBES5OFIPW6ET2CXSQ, seg_pos={0 0}, upload=506314]: Long tail cancellation due to optimal threshold Dec 29 17:03:25.334653125 (577h34m25.20573618s) [noise=true , address=77.90.31.18:4805 , piece_id=2DTKHJDUXOHD4NKINQCMKF6M3WPUCYJTMZLPYH6XNYWNWOS6KZXQ, seg_pos={0 0}, upload=506314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 17:03:25.334661541 (577h34m25.205744576s) [noise=true , address=77.90.31.18:4805 , piece_id=2DTKHJDUXOHD4NKINQCMKF6M3WPUCYJTMZLPYH6XNYWNWOS6KZXQ, seg_pos={0 0}, upload=506314]: Long tail cancellation due to optimal threshold Dec 29 17:03:25.334685336 (577h34m25.205768371s) [seg_pos={0 0}, upload=506314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:03:25.334775547 (577h34m25.205858592s) [upload=506314]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:03:25.334781880 (577h34m25.205864925s) [upload=506314]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:03:25.633187479 (577h34m25.504270514s) [upload=506314]: Done uploading segments... Dec 29 17:03:25.633208299 (577h34m25.504291354s) [upload=506314]: Upload finished. err: <nil> Duration: 30.002968866s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XXX] duration: 5.001638262s trace: https://jaeger.dev.storj.io/trace/27858c58fa347e97 FAIL (2025-12-29 17:03:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001664802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......X......................X.......] duration: 37.904791568s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 17:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 37.904819151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX..] duration: 5.770605724s trace: https://jaeger.dev.storj.io/trace/3c8fe8e9719880ca FAIL (2025-12-29 17:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q) context deadline exceeded ======LOGS====== 9 17:00:45.539911988 (577h31m45.410995023s) [upload=506280]: Starting upload Dec 29 17:00:45.539986039 (577h31m45.411069074s) [upload=506280]: Done starting upload Dec 29 17:00:45.540046224 (577h31m45.411129279s) [upload=506280]: Uploading segments... Dec 29 17:00:45.540180901 (577h31m45.411263936s) [upload=506280]: Got next segment. Inline: false Dec 29 17:00:45.540216719 (577h31m45.411299754s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:00:45.540221097 (577h31m45.411304142s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:00:45.789919108 (577h31m45.661002143s) [seg_pos={0 0}, upload=506280]: Begin upload segment... Dec 29 17:00:45.789961138 (577h31m45.661044173s) [seg_pos={0 0}, upload=506280]: Done begin upload segment. Dec 29 17:00:45.789987519 (577h31m45.661070564s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790033055 (577h31m45.6611161s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790546112 (577h31m45.661629147s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790949070 (577h31m45.662032105s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791365723 (577h31m45.662448768s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791818365 (577h31m45.66290141s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792248674 (577h31m45.663331719s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792710193 (577h31m45.663793238s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.794496304 (577h31m45.66557934s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.795684126 (577h31m45.666767171s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.797240760 (577h31m45.668323805s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.800093171 (577h31m45.671176206s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.801977329 (577h31m45.673060374s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.804772762 (577h31m45.675855808s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.806558573 (577h31m45.677641608s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.811097495 (577h31m45.68218053s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:47.489597096 (577h31m47.360680131s) [upload=506280]: Next returned nil segment Dec 29 17:00:47.489610752 (577h31m47.360693787s) [upload=506280]: Waiting for error group managing segments... Dec 29 17:00:47.769739099 (577h31m47.640822144s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.770748601 (577h31m47.641831646s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785560665 (577h31m47.65664371s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785772348 (577h31m47.656855383s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.799070519 (577h31m47.670153564s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.801725265 (577h31m47.67280831s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.803057211 (577h31m47.674140256s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.814788328 (577h31m47.685871373s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.922372444 (577h31m47.793455479s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.046216582 (577h31m47.917299627s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.051177748 (577h31m47.922260784s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.057471553 (577h31m47.928554598s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229734237 (577h31m48.100817282s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229759335 (577h31m48.10084238s) [seg_pos={0 0}, upload=506280]: Segment reached optimal threshold of 13 pieces. Dec 29 17:00:48.230163884 (577h31m48.101246939s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230174205 (577h31m48.10125726s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230537917 (577h31m48.101620952s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230547996 (577h31m48.101631031s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230712549 (577h31m48.101795594s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection Dec 29 17:00:48.230722859 (577h31m48.101805904s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230747917 (577h31m48.101830952s) [seg_pos={0 0}, upload=506280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:00:48.230825324 (577h31m48.101908369s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:00:48.230831486 (577h31m48.101914531s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:00:48.532400788 (577h31m48.403483843s) [upload=506280]: Done uploading segments... Dec 29 17:00:48.532425275 (577h31m48.40350831s) [upload=506280]: Upload finished. err: <nil> Duration: 5.770637053s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.XX] duration: 5.000960874s trace: https://jaeger.dev.storj.io/trace/57766469baac2495 FAIL (2025-12-29 17:03:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001208165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......X......................X......] duration: 31.166084612s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 17:02:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 31.166115621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [.XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.X] duration: 5.001074369s trace: https://jaeger.dev.storj.io/trace/1e45f5bb914ecfdd FAIL (2025-12-29 17:02:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001099367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.XXX....X...XXXXX.X..XX...XX..XXXXXXX.XXXX.XXXX.XX.X....XXX.] duration: 8.895001563s trace: https://jaeger.dev.storj.io/trace/3c8fe8e9719880ca FAIL (2025-12-29 17:02:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q) context deadline exceeded ======LOGS====== 9 17:00:45.539911988 (577h31m45.410995023s) [upload=506280]: Starting upload Dec 29 17:00:45.539986039 (577h31m45.411069074s) [upload=506280]: Done starting upload Dec 29 17:00:45.540046224 (577h31m45.411129279s) [upload=506280]: Uploading segments... Dec 29 17:00:45.540180901 (577h31m45.411263936s) [upload=506280]: Got next segment. Inline: false Dec 29 17:00:45.540216719 (577h31m45.411299754s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 17:00:45.540221097 (577h31m45.411304142s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 17:00:45.789919108 (577h31m45.661002143s) [seg_pos={0 0}, upload=506280]: Begin upload segment... Dec 29 17:00:45.789961138 (577h31m45.661044173s) [seg_pos={0 0}, upload=506280]: Done begin upload segment. Dec 29 17:00:45.789987519 (577h31m45.661070564s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790033055 (577h31m45.6611161s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790546112 (577h31m45.661629147s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.790949070 (577h31m45.662032105s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791365723 (577h31m45.662448768s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.791818365 (577h31m45.66290141s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792248674 (577h31m45.663331719s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.792710193 (577h31m45.663793238s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.794496304 (577h31m45.66557934s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.795684126 (577h31m45.666767171s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.797240760 (577h31m45.668323805s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.800093171 (577h31m45.671176206s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.801977329 (577h31m45.673060374s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.804772762 (577h31m45.675855808s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.806558573 (577h31m45.677641608s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:45.811097495 (577h31m45.68218053s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Uploading piece... Dec 29 17:00:47.489597096 (577h31m47.360680131s) [upload=506280]: Next returned nil segment Dec 29 17:00:47.489610752 (577h31m47.360693787s) [upload=506280]: Waiting for error group managing segments... Dec 29 17:00:47.769739099 (577h31m47.640822144s) [noise=true , address=145.79.227.8:4329 , piece_id=AUMS5RFNJA3NAYHS6AUVDGC3TO4ZFGMWHKESNGBVBEDVZ73JF7DQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.770748601 (577h31m47.641831646s) [noise=true , address=145.79.227.15:4873 , piece_id=76NDNBEWTROGTCNPBG56AZUSPNGC33V7UHDCC2OMD322JVKIWNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785560665 (577h31m47.65664371s) [noise=true , address=145.79.227.50:4373 , piece_id=AIWX2TOHKOHPAX6YW7EZ2E6CNV2HM45CPWJHLGWBZMLCHQPFCYLA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.785772348 (577h31m47.656855383s) [noise=true , address=145.79.227.3:4151 , piece_id=XFREHRJ7A6YKASFUFASAWKKEDOORPGMHU6TVPCDPDT66CHUUCSRQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.799070519 (577h31m47.670153564s) [noise=true , address=145.79.227.22:4509 , piece_id=AR72NLFK6NOHS6TOYSZML673GORGS3X5Z72MVP2VPAAU34AKV4YA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.801725265 (577h31m47.67280831s) [noise=true , address=145.79.227.16:4910 , piece_id=OZYALE4IHKIQ4MZGMER6QXPX6DFZAD4IOK3C5M7L2KM5IIQZRCOQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.803057211 (577h31m47.674140256s) [noise=true , address=145.79.227.26:4653 , piece_id=UH3RSFNA6I4R4DHBXRC7266TSNPVFP5FBY4JOD56OGK4Z7ZEFNLQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.814788328 (577h31m47.685871373s) [noise=true , address=145.79.227.42:4123 , piece_id=OAOIID5WANPGNHPWOVGSJJSZ5W2PTOFPE65EPINUNT6UMV5SE2UQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:47.922372444 (577h31m47.793455479s) [noise=true , address=145.79.227.18:4006 , piece_id=KGCVAYT6UEB2Y6CXM33F6NQYZRJB2YQ4B53CQZ7QK7JVK627TTIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.046216582 (577h31m47.917299627s) [noise=true , address=45.91.116.9:4264 , piece_id=QTCX43TADOELBPYD35YIRFV242O2HD2RUANNYFC7VI2NTTGXDTYQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.051177748 (577h31m47.922260784s) [noise=true , address=45.91.116.4:4940 , piece_id=6437A5YQW6Y2INQM66AFSBGHO45YLNYB7QYO3R6EW4KKD4JMRPIQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.057471553 (577h31m47.928554598s) [noise=true , address=45.91.116.11:4117 , piece_id=ENPB77DGLF2MVGBKO64SON7FDY4IEI4GWDKVY4TMBFESJGV5MV2Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229734237 (577h31m48.100817282s) [noise=true , address=145.79.227.41:4085 , piece_id=54A7KBP7QJQ5RYXSN4WLF4H6FH27XDO6HGVGSKXYDDELGUDF4X4Q, seg_pos={0 0}, upload=506280]: Done uploading piece. err: <nil> Dec 29 17:00:48.229759335 (577h31m48.10084238s) [seg_pos={0 0}, upload=506280]: Segment reached optimal threshold of 13 pieces. Dec 29 17:00:48.230163884 (577h31m48.101246939s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230174205 (577h31m48.10125726s) [noise=true , address=45.91.116.14:4192 , piece_id=3AFSDBDEB7JVACEOZU3D552GMKZNEKL7R3ITAOQRSG6RL6CMGNGA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230537917 (577h31m48.101620952s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 17:00:48.230547996 (577h31m48.101631031s) [noise=true , address=145.79.227.40:4050 , piece_id=DYVGEHQJRGYAWFWON5KKRFMCWABWQ7EGWOPQKHP3ZXMCIY5MGRHA, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230712549 (577h31m48.101795594s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45346->145.79.227.17:4971: use of closed network connection Dec 29 17:00:48.230722859 (577h31m48.101805904s) [noise=true , address=145.79.227.17:4971 , piece_id=ZMPPPPHDPT2XTRPZQUMJUW27GREZUO2PJIJCZO75IGIDQZ66ORCQ, seg_pos={0 0}, upload=506280]: Long tail cancellation due to optimal threshold Dec 29 17:00:48.230747917 (577h31m48.101830952s) [seg_pos={0 0}, upload=506280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 17:00:48.230825324 (577h31m48.101908369s) [upload=506280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 17:00:48.230831486 (577h31m48.101914531s) [upload=506280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 17:00:48.532400788 (577h31m48.403483843s) [upload=506280]: Done uploading segments... Dec 29 17:00:48.532425275 (577h31m48.40350831s) [upload=506280]: Upload finished. err: <nil> Duration: 8.895028956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X......................X.....] duration: 37.445721419s trace: https://jaeger.dev.storj.io/trace/741bec3beb820041 FAIL (2025-12-29 17:01:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381647614 (577h27m21.252730659s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.381975108 (577h27m21.253058143s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382203963 (577h27m21.253286998s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382511739 (577h27m21.253594774s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382585990 (577h27m21.253669035s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.382962027 (577h27m21.254045072s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383240297 (577h27m21.254323342s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383425028 (577h27m21.254508063s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383667890 (577h27m21.254750926s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.383826273 (577h27m21.254909318s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384001296 (577h27m21.255084361s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.384471662 (577h27m21.255554707s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385037539 (577h27m21.256120574s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:21.385498116 (577h27m21.256581151s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Uploading piece... Dec 29 16:56:22.841124503 (577h27m22.712207538s) [upload=506219]: Next returned nil segment Dec 29 16:56:22.841137738 (577h27m22.712220783s) [upload=506219]: Waiting for error group managing segments... Dec 29 16:56:23.254662209 (577h27m23.125745254s) [noise=true , address=145.79.227.10:4435 , piece_id=GWBODI3LKIJVO6NUFW2SPY4X7QKHSM6KN7W6LK3TR6ZQYFPZD3UA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.258874640 (577h27m23.129957685s) [noise=true , address=145.79.227.20:4454 , piece_id=ITRIKSTSAIUK2WSXLS5T4AC7LZI5WXYUQ7PQ5AUAL24ZDVGEJWFQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.263770463 (577h27m23.134853508s) [noise=true , address=145.79.227.47:4299 , piece_id=PO5YWOZTNGU2EAYLMUYGALUCBZJEC5ZID7MX4EMTWIR5MCWAPB6A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.265640875 (577h27m23.13672392s) [noise=true , address=145.79.227.51:4433 , piece_id=QHSFV775HSYV6XRG636MKEWUXLKBO4Q44ZAOXCAENLVBHGV5LZPQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.270053166 (577h27m23.141136211s) [noise=true , address=145.79.227.37:4584 , piece_id=GGRI76SJZ54VEOU77MMMZDT6U5WT47MJK3GGZVPUMT3UCJKMZI3Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.516579067 (577h27m23.387662102s) [noise=true , address=77.90.31.10:4523 , piece_id=QFRMI33YQ4F2TECIVVKESE3GLL62DZ7XALG5ZHI7IUJRX4NKR7TQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:23.891608979 (577h27m23.762692014s) [noise=true , address=145.79.227.12:4788 , piece_id=6TDJAHDUBBTHEEGGM6ZPJ2MXRBFTDLJN2X2EQO7SEWRZ46WFRR5A, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.119353067 (577h27m23.990436112s) [noise=true , address=145.79.227.44:4208 , piece_id=WHHDF5LYESUWEOYU2QWWOGPOLW5IW3DYNLIGP2I346ESJWYOPTIA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.150579536 (577h27m24.021662571s) [noise=true , address=145.79.227.43:4186 , piece_id=PYLNP6ERLC6PTHDZSKJ4I3OCEO5MMI64TTJKF6WUTCMNX2XVMV4Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.174696108 (577h27m24.045779143s) [noise=true , address=145.79.227.33:4900 , piece_id=KQT4PASKPG33KATU2DOHQQRHSXKOREUOWV7SZLBHGPGWGVD5KVYA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.204879823 (577h27m24.075962868s) [noise=true , address=145.79.227.40:4070 , piece_id=AB7VKUZB75MXAY6VUOP3HUWC3PPHMCD2ZVC7VKSFZUTQGNRVAYGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.225611097 (577h27m24.096694132s) [noise=true , address=77.90.31.14:4669 , piece_id=QWCPDX2WLUDZJM3XTNEX2XQ2N22RBE4MHBTJWENGVYWYYSPYDVKA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.227033616 (577h27m24.098116661s) [noise=true , address=145.79.227.44:4203 , piece_id=2B3YYQX2T323S32NMAWFXIZBGDSJYQOHLF2QG7PEH2BCAZ7DGAXQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256699715 (577h27m24.12778275s) [noise=true , address=77.90.31.21:4909 , piece_id=GAHR2C4CN6GOTZ2EOTERG3Q7EWWBO6QH6SE4UA2SQLUOSI6N7DGA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: <nil> Dec 29 16:56:24.256722037 (577h27m24.127805072s) [seg_pos={0 0}, upload=506219]: Segment reached optimal threshold of 14 pieces. Dec 29 16:56:24.256962816 (577h27m24.128045861s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.256970009 (577h27m24.128053044s) [noise=true , address=145.79.227.34:4920 , piece_id=LN7IYDYQ5I4NDGYDZ6TLZRBAZWKKYBLB62VCQHJBEU3FJBRSY7JA, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257284197 (577h27m24.128367243s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44670->64.49.13.169:4705: use of closed network connection Dec 29 16:56:24.257292344 (577h27m24.128375389s) [noise=true , address=64.49.13.169:4705 , piece_id=X6JM2UAP5WZW6OKZONZNK6JBYUQQDTG2FAIO7LN3RWXZ3G2YB5PQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257414957 (577h27m24.128497992s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk): protocol: expected piece hash; context canceled; EOF Dec 29 16:56:24.257421178 (577h27m24.128504213s) [noise=true , address=145.79.227.35:4514 , piece_id=75N4JK2ECAYS7RFWCPAPYUWDETX6DZ4MEJUQTAKZPZCJ44SWAE6Q, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257514566 (577h27m24.128597612s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51536->64.49.13.169:4707: use of closed network connection Dec 29 16:56:24.257520828 (577h27m24.128603873s) [noise=true , address=64.49.13.169:4707 , piece_id=442N4MQH5G6VUJGIRAUF5LEWZ3AEZXQQPNTRYYGZ22RDOEAMMCBQ, seg_pos={0 0}, upload=506219]: Long tail cancellation due to optimal threshold Dec 29 16:56:24.257535346 (577h27m24.128618382s) [seg_pos={0 0}, upload=506219]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:56:24.257606432 (577h27m24.128689467s) [upload=506219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:56:24.257612062 (577h27m24.128695117s) [upload=506219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:56:24.564524197 (577h27m24.435607232s) [upload=506219]: Done uploading segments... Dec 29 16:56:24.564542192 (577h27m24.435625227s) [upload=506219]: Upload finished. err: <nil> Duration: 37.445740014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XXX.X.X.X.XXX........X.] duration: 5.00155668s trace: https://jaeger.dev.storj.io/trace/176351fc92e5fa36 FAIL (2025-12-29 17:01:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001584113s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)