Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...........X.....................X.....X..XX...........] duration: 46.587376756s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:43:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 46.587406663s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XX] duration: 5.000477661s trace: https://jaeger.dev.storj.io/trace/18446212b3c8c1ca FAIL (2026-01-01 07:43:25 UTC) orbiter warning: context deadline exceeded Duration: 5.000505714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X.........X] duration: 30.00336263s trace: https://jaeger.dev.storj.io/trace/49d044a496f36a33 FAIL (2026-01-01 07:43:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MLHW4H5FAZHE3S3YDLUHTONHUFAM5HZWDGHANZZAXP2WBSKJ2XZA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: PXNQAAZZUZMXNZKQJ5ZSKDBBNRA7EZ4GEBEP77SQY7B3CE7R5FYA) context deadline exceeded ======LOGS====== Jan 1 07:42:47.068223081 (640h13m46.939306116s) [upload=560984]: Starting upload Jan 1 07:42:47.068282043 (640h13m46.939365078s) [upload=560984]: Done starting upload Jan 1 07:42:47.068332039 (640h13m46.939415074s) [upload=560984]: Uploading segments... Jan 1 07:42:47.068498616 (640h13m46.939581651s) [upload=560984]: Got next segment. Inline: false Jan 1 07:42:47.068526999 (640h13m46.939610044s) [upload=560984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:42:47.068531598 (640h13m46.939614633s) [upload=560984]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:42:47.311919482 (640h13m47.183002527s) [seg_pos={0 0}, upload=560984]: Begin upload segment... Jan 1 07:42:47.311970358 (640h13m47.183053403s) [seg_pos={0 0}, upload=560984]: Done begin upload segment. Jan 1 07:42:47.312001708 (640h13m47.183084743s) [noise=true , address=145.79.227.11:4731 , piece_id=3NDFXG7GUD7BQPHMX7UP4LTS7D66IR5XR76GT3M7UY5TRMEESJ7Q, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.312043197 (640h13m47.183126242s) [noise=true , address=145.79.227.42:4121 , piece_id=2ADSBIS4HR7WIZKCNH2J5XI5EFEKQKH6AS5YZG26RD6EOWOUH7PQ, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.312630826 (640h13m47.183713871s) [noise=true , address=145.79.227.38:4977 , piece_id=NOM5UYP2B6M52N3EMYWVCNFEVRFOEAACXZCOGXQKAV2VENRB7YLA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.312694277 (640h13m47.183777322s) [noise=true , address=145.79.227.36:4517 , piece_id=POIFRT3BU72GPEBYK7M6LLUIXQK2WI3RJJB4IARKNZPJ7N4XCS4Q, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.313080702 (640h13m47.184163747s) [noise=true , address=145.79.227.26:4654 , piece_id=7XIGZ5HNFOZPUPZECDBNMATOTYK73DIJFD6P6BV4VIZPDJ36ET2A, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.313443844 (640h13m47.18452689s) [noise=true , address=145.79.227.27:4689 , piece_id=VLH2NTUL7C2LHYKOGZYI66YU7BHBDXCTLJDK3IQN7MHSBOQOLKRQ, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.313778381 (640h13m47.184861426s) [noise=true , address=45.91.116.4:4937 , piece_id=UY32TG33EUTXZPIHPVIL3GVHZNHJOTH5V6W4ZUHL2ZD5FGQJMMXA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.313876618 (640h13m47.184959663s) [noise=true , address=145.79.227.19:4077 , piece_id=Y26EYBACORO245AWV4DIG25Y3KBLXYZWG5NOEOA5YWQ322V332ZA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.314247575 (640h13m47.18533062s) [noise=true , address=77.90.31.17:4773 , piece_id=PXNQAAZZUZMXNZKQJ5ZSKDBBNRA7EZ4GEBEP77SQY7B3CE7R5FYA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.314377231 (640h13m47.185460266s) [noise=true , address=145.79.227.15:4875 , piece_id=7YC7IZZ26LKF2BWWUO6NZ6TTDXBHJ4GJRT2OKOPVOUU5RCU6PDOQ, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.314605676 (640h13m47.185688721s) [noise=true , address=77.90.31.14:4663 , piece_id=YF66CU4TJKF4CJOWOY65UF4UGQY3EWWU3FHAVYD7CVDNQWSOIZPA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.314862565 (640h13m47.1859456s) [noise=true , address=145.79.227.50:4373 , piece_id=IQETQC2CRKOPEY2R6KX47HNGJLE4F7J5OAFLO3VYMZ4OLHZSP6SA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.314961033 (640h13m47.186044068s) [noise=true , address=145.79.227.46:4601 , piece_id=5G63VVQWGJC3MCI4IZD3MP2VUZRWK5UMQGEIVDHXNIYKUYRB6XJQ, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.315379169 (640h13m47.186462214s) [noise=true , address=77.90.31.11:4554 , piece_id=BZ4XU26LIYCDEEFUSC2PPKH62LFQAOB5JQ4KXLDDKZK53WOOYHJA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.315885223 (640h13m47.186968258s) [noise=true , address=77.90.31.12:4589 , piece_id=MLHW4H5FAZHE3S3YDLUHTONHUFAM5HZWDGHANZZAXP2WBSKJ2XZA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:47.316420662 (640h13m47.187503697s) [noise=true , address=145.79.227.18:4007 , piece_id=ZRNBGDAU3UN5ZV5Z5EKK7YI5T7DSPB25BB75YGFYP5PQYTGDRTTA, seg_pos={0 0}, upload=560984]: Uploading piece... Jan 1 07:42:49.023975691 (640h13m48.895058727s) [upload=560984]: Next returned nil segment Jan 1 07:42:49.023988246 (640h13m48.895071281s) [upload=560984]: Waiting for error group managing segments... Jan 1 07:42:49.301859138 (640h13m49.172942183s) [noise=true , address=145.79.227.26:4654 , piece_id=7XIGZ5HNFOZPUPZECDBNMATOTYK73DIJFD6P6BV4VIZPDJ36ET2A, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.302796894 (640h13m49.173879939s) [noise=true , address=145.79.227.11:4731 , piece_id=3NDFXG7GUD7BQPHMX7UP4LTS7D66IR5XR76GT3M7UY5TRMEESJ7Q, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.306102328 (640h13m49.177185383s) [noise=true , address=145.79.227.36:4517 , piece_id=POIFRT3BU72GPEBYK7M6LLUIXQK2WI3RJJB4IARKNZPJ7N4XCS4Q, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.310780606 (640h13m49.181863651s) [noise=true , address=145.79.227.50:4373 , piece_id=IQETQC2CRKOPEY2R6KX47HNGJLE4F7J5OAFLO3VYMZ4OLHZSP6SA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.311642327 (640h13m49.182725372s) [noise=true , address=145.79.227.18:4007 , piece_id=ZRNBGDAU3UN5ZV5Z5EKK7YI5T7DSPB25BB75YGFYP5PQYTGDRTTA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.313110292 (640h13m49.184193327s) [noise=true , address=145.79.227.27:4689 , piece_id=VLH2NTUL7C2LHYKOGZYI66YU7BHBDXCTLJDK3IQN7MHSBOQOLKRQ, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.319964984 (640h13m49.191048039s) [noise=true , address=145.79.227.19:4077 , piece_id=Y26EYBACORO245AWV4DIG25Y3KBLXYZWG5NOEOA5YWQ322V332ZA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.323793685 (640h13m49.19487672s) [noise=true , address=145.79.227.42:4121 , piece_id=2ADSBIS4HR7WIZKCNH2J5XI5EFEKQKH6AS5YZG26RD6EOWOUH7PQ, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.385348573 (640h13m49.256431608s) [noise=true , address=145.79.227.15:4875 , piece_id=7YC7IZZ26LKF2BWWUO6NZ6TTDXBHJ4GJRT2OKOPVOUU5RCU6PDOQ, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.385884895 (640h13m49.25696794s) [noise=true , address=145.79.227.46:4601 , piece_id=5G63VVQWGJC3MCI4IZD3MP2VUZRWK5UMQGEIVDHXNIYKUYRB6XJQ, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.399171053 (640h13m49.270254088s) [noise=true , address=145.79.227.38:4977 , piece_id=NOM5UYP2B6M52N3EMYWVCNFEVRFOEAACXZCOGXQKAV2VENRB7YLA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.573213052 (640h13m49.444296098s) [noise=true , address=77.90.31.17:4773 , piece_id=PXNQAAZZUZMXNZKQJ5ZSKDBBNRA7EZ4GEBEP77SQY7B3CE7R5FYA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.579364064 (640h13m49.450447109s) [noise=true , address=77.90.31.12:4589 , piece_id=MLHW4H5FAZHE3S3YDLUHTONHUFAM5HZWDGHANZZAXP2WBSKJ2XZA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: <nil> Jan 1 07:42:49.579389012 (640h13m49.450472047s) [seg_pos={0 0}, upload=560984]: Segment reached optimal threshold of 13 pieces. Jan 1 07:42:49.579646041 (640h13m49.450729076s) [noise=true , address=77.90.31.11:4554 , piece_id=BZ4XU26LIYCDEEFUSC2PPKH62LFQAOB5JQ4KXLDDKZK53WOOYHJA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Jan 1 07:42:49.579653185 (640h13m49.45073623s) [noise=true , address=77.90.31.11:4554 , piece_id=BZ4XU26LIYCDEEFUSC2PPKH62LFQAOB5JQ4KXLDDKZK53WOOYHJA, seg_pos={0 0}, upload=560984]: Long tail cancellation due to optimal threshold Jan 1 07:42:49.579819762 (640h13m49.450902797s) [noise=true , address=77.90.31.14:4663 , piece_id=YF66CU4TJKF4CJOWOY65UF4UGQY3EWWU3FHAVYD7CVDNQWSOIZPA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jan 1 07:42:49.579853546 (640h13m49.450936581s) [noise=true , address=77.90.31.14:4663 , piece_id=YF66CU4TJKF4CJOWOY65UF4UGQY3EWWU3FHAVYD7CVDNQWSOIZPA, seg_pos={0 0}, upload=560984]: Long tail cancellation due to optimal threshold Jan 1 07:42:49.579978845 (640h13m49.45106188s) [noise=true , address=45.91.116.4:4937 , piece_id=UY32TG33EUTXZPIHPVIL3GVHZNHJOTH5V6W4ZUHL2ZD5FGQJMMXA, seg_pos={0 0}, upload=560984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Jan 1 07:42:49.579987401 (640h13m49.451070436s) [noise=true , address=45.91.116.4:4937 , piece_id=UY32TG33EUTXZPIHPVIL3GVHZNHJOTH5V6W4ZUHL2ZD5FGQJMMXA, seg_pos={0 0}, upload=560984]: Long tail cancellation due to optimal threshold Jan 1 07:42:49.580004664 (640h13m49.451087699s) [seg_pos={0 0}, upload=560984]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:42:49.580070780 (640h13m49.451153825s) [upload=560984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:42:49.580079606 (640h13m49.451162671s) [upload=560984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:42:49.873059600 (640h13m49.744142646s) [upload=560984]: Done uploading segments... Jan 1 07:42:49.873084729 (640h13m49.744167764s) [upload=560984]: Upload finished. err: <nil> Duration: 30.003383849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...........X.....................X.....X..XX..........] duration: 35.865704477s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:42:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 35.865768187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.X] duration: 5.001495926s trace: https://jaeger.dev.storj.io/trace/2bb3473c13d3dfe0 FAIL (2026-01-01 07:42:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001529871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X.........] duration: 8.658215197s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:42:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 8.658245166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...........X.....................X.....X..XX.........] duration: 37.649587176s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:41:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 37.649609608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.] duration: 3.638404699s trace: https://jaeger.dev.storj.io/trace/cc42fb0ae8bda34 FAIL (2026-01-01 07:41:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.638447811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX] duration: 5.000518903s trace: https://jaeger.dev.storj.io/trace/cc42fb0ae8bda34 FAIL (2026-01-01 07:41:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000542799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X........] duration: 5.796177751s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:41:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 5.796208489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX..X] duration: 5.000795832s trace: https://jaeger.dev.storj.io/trace/1429df6448fe590a FAIL (2026-01-01 07:41:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000820269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X...........X.....................X.....X..XX........] duration: 19.820207506s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:40:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 19.820260055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X...........X.....................X.....X..XX.......] duration: 38.717267018s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:40:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 38.717293558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX..] duration: 4.70952829s trace: https://jaeger.dev.storj.io/trace/6b6971490d7cda62 FAIL (2026-01-01 07:40:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.709552168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X.......] duration: 5.787030225s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:40:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 5.787057686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X......] duration: 6.171625034s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:39:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 6.171683425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X...........X.....................X.....X..XX......] duration: 25.261626922s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:39:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 25.26164694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX.] duration: 3.592156808s trace: https://jaeger.dev.storj.io/trace/6b6971490d7cda62 FAIL (2026-01-01 07:39:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592187004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X...........X.....................X.....X..XX.....] duration: 39.845949652s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:39:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 39.845967535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXXX] duration: 5.001352482s trace: https://jaeger.dev.storj.io/trace/6b6971490d7cda62 FAIL (2026-01-01 07:38:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001375756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X.....] duration: 8.26448835s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:38:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 8.264519388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X...........X.....................X.....X..XX....] duration: 35.106555228s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:38:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 35.10657171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X....] duration: 6.031997477s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:38:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 6.032023737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XXX] duration: 5.000849948s trace: https://jaeger.dev.storj.io/trace/68cd351c0299f226 FAIL (2026-01-01 07:38:05 UTC) orbiter warning: context deadline exceeded Duration: 5.00088784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X...........X.....................X.....X..XX...] duration: 39.429443754s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:37:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 39.42945794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [X.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.XX] duration: 5.001634721s trace: https://jaeger.dev.storj.io/trace/387792de31f8a881 FAIL (2026-01-01 07:37:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001660711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XX.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.X] duration: 5.001186765s trace: https://jaeger.dev.storj.io/trace/4b201652eefc6513 FAIL (2026-01-01 07:37:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001225379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X...] duration: 5.910787769s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 5.910816283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X............X...........X.....................X.....X..XX..] duration: 36.640668562s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:37:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 36.640713979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X..] duration: 5.835158125s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:36:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 5.835187121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X.] duration: 5.655411172s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:36:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 5.655448683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXX.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX.] duration: 3.538733316s trace: https://jaeger.dev.storj.io/trace/4bd941d189ce2923 FAIL (2026-01-01 07:36:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538769904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...X] duration: 30.002818952s trace: https://jaeger.dev.storj.io/trace/2ffce02da305d7ac FAIL (2026-01-01 07:35:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 640h6m28.792433757s) [upload=560881]: Uploading segments... Jan 1 07:35:28.921464618 (640h6m28.792547653s) [upload=560881]: Got next segment. Inline: false Jan 1 07:35:28.921541214 (640h6m28.792624259s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:35:28.921547075 (640h6m28.79263011s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:35:29.170634673 (640h6m29.041717708s) [seg_pos={0 0}, upload=560881]: Begin upload segment... Jan 1 07:35:29.170693045 (640h6m29.0417761s) [seg_pos={0 0}, upload=560881]: Done begin upload segment. Jan 1 07:35:29.170724674 (640h6m29.041807709s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170767707 (640h6m29.041850752s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.170885661 (640h6m29.041968696s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171382086 (640h6m29.042465121s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.171827654 (640h6m29.042910699s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172261130 (640h6m29.043344185s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.172688304 (640h6m29.043771339s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.173184719 (640h6m29.044267764s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.174882832 (640h6m29.045965877s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.176678561 (640h6m29.047761606s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.178407063 (640h6m29.049490108s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.179209451 (640h6m29.050292486s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.180310747 (640h6m29.051393792s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.181496194 (640h6m29.052579229s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.182170038 (640h6m29.053253073s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:29.183260815 (640h6m29.05434385s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Uploading piece... Jan 1 07:35:30.886612294 (640h6m30.757695329s) [upload=560881]: Next returned nil segment Jan 1 07:35:30.886625108 (640h6m30.757708143s) [upload=560881]: Waiting for error group managing segments... Jan 1 07:35:31.169989033 (640h6m31.041072068s) [noise=true , address=145.79.227.45:4230 , piece_id=OOU6EKNMLDARHBCJSZ3CW5F6WRJRKQN3FQTGV5CH3UQW6J3YNNMQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.170744441 (640h6m31.041827486s) [noise=true , address=145.79.227.11:4729 , piece_id=5HJKM3KB62XWCTRFIONB5JLTF7H6EPRWY6VTSTMIF6NS7QPSYW5Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.171415750 (640h6m31.042498795s) [noise=true , address=145.79.227.37:4565 , piece_id=JGIGEKIW7I63G56RCHBJJVMLEHCW6WSYXPEW64LEGH4G6X4S3H2Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.172987603 (640h6m31.044070638s) [noise=true , address=145.79.227.41:4087 , piece_id=GSCXH732GIFBBV4JZ54BG4JTBMPF32T3ZGPHN3VNZHXAXSB4KFCA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.194040310 (640h6m31.065123355s) [noise=true , address=145.79.227.5:4222 , piece_id=T7OP24AP7IRP6P277LHSBQCMTLJRBTO2GOWW4JATCB7GJEEDQUUA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.199571402 (640h6m31.070654447s) [noise=true , address=145.79.227.22:4510 , piece_id=AKBLYWXDEBTF5V3F2TLUJ574QSDKVU52NHWBLLYERG2HOGC22YGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.256964225 (640h6m31.12804726s) [noise=true , address=145.79.227.3:4151 , piece_id=7DXSA5IHLOQIPQHQCATHZFKJEV32MI5CKSXU2ZOOZQYP4AFRRD4A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.441406871 (640h6m31.312489916s) [noise=true , address=77.90.31.19:4842 , piece_id=AHGIBX26B7MCVKWTU4PLWOXINEQGHJWG7P674L5XYWDFTBZWHT7A, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.478855457 (640h6m31.349938502s) [noise=true , address=77.90.31.11:4556 , piece_id=RNNTR2I4MJIRORFUJXUMKNCAGGW4DCWZGCKE4SDPXANT5JGLJFUQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.482153618 (640h6m31.353236653s) [noise=true , address=77.90.31.16:4735 , piece_id=M2PPJIPCUH4GIX3UYXBRQ3QLXYEG4YMSR5SZGLFA5DZUJ4SPF2GA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.626631484 (640h6m31.497714529s) [noise=true , address=45.91.116.14:4192 , piece_id=CVMTH3MH3NM6QR3W6U24ENAJVTG3SYPIQL6GLBDJW4OW4DGUBQIQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.754146531 (640h6m31.625229566s) [noise=true , address=45.91.116.13:4154 , piece_id=HSJUQ4FZTJUMRIILWFYKV3XWOUA7ABABURYMLFTJ36CPZ4I2OVNQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778102947 (640h6m31.649185992s) [noise=true , address=45.91.116.9:4263 , piece_id=EQB55SPOONXMXEBTVULXKMPIP7W5IWQKY5FOSPRM4JPT4F54COGA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: <nil> Jan 1 07:35:31.778120690 (640h6m31.649203725s) [seg_pos={0 0}, upload=560881]: Segment reached optimal threshold of 13 pieces. Jan 1 07:35:31.778295743 (640h6m31.649378788s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->45.91.116.4:4937: use of closed network connection Jan 1 07:35:31.778301445 (640h6m31.64938448s) [noise=true , address=45.91.116.4:4937 , piece_id=KSSRYRJVQ4LAHFS6XEMOBCKXIIJQW44SID2LKWQ5Z7SURCDNIWAA, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778557643 (640h6m31.649640678s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36670->45.91.116.11:4119: use of closed network connection Jan 1 07:35:31.778563914 (640h6m31.649646949s) [noise=true , address=45.91.116.11:4119 , piece_id=VLBBQJBR3QX7IXYUC4NPLF634F3E2OXOVOS3RHHD3MH4XNRJLVGQ, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778721655 (640h6m31.64980469s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:35:31.778772812 (640h6m31.649855847s) [noise=true , address=145.79.227.33:4871 , piece_id=O4DIOUVZU3CUMCKGVXMMVRIGN6MSPEO56VWZNCF7G4665YKU2D6Q, seg_pos={0 0}, upload=560881]: Long tail cancellation due to optimal threshold Jan 1 07:35:31.778797339 (640h6m31.649880384s) [seg_pos={0 0}, upload=560881]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:35:31.779402000 (640h6m31.650485035s) [upload=560881]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:35:31.779409073 (640h6m31.650492108s) [upload=560881]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:35:32.074341526 (640h6m31.945424581s) [upload=560881]: Done uploading segments... Jan 1 07:35:32.074368638 (640h6m31.945451673s) [upload=560881]: Upload finished. err: <nil> Duration: 30.002846965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX............X...........X.....................X.....X..XX.] duration: 15.426226615s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:35:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 15.426246062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXX.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXXX] duration: 5.000749426s trace: https://jaeger.dev.storj.io/trace/4bd941d189ce2923 FAIL (2026-01-01 07:35:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000769013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX............X...........X.....................X.....X..XX] duration: 59.335932062s trace: https://jaeger.dev.storj.io/trace/4bd4247e4ec7de08 FAIL (2026-01-01 07:35:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== }, upload=560864]: Uploading piece... Jan 1 07:34:23.356901112 (640h5m23.227984157s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357300553 (640h5m23.228383599s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.357780026 (640h5m23.228863071s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358509805 (640h5m23.22959285s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.358982947 (640h5m23.230065992s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359467550 (640h5m23.230550605s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.359870576 (640h5m23.230953621s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360248336 (640h5m23.231331381s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.360668456 (640h5m23.231751491s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361212272 (640h5m23.232295307s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.361639635 (640h5m23.23272268s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362186146 (640h5m23.233269191s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.362615954 (640h5m23.233698999s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363116197 (640h5m23.234199232s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.363751637 (640h5m23.234834672s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364249605 (640h5m23.23533264s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:23.364858093 (640h5m23.235941138s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Uploading piece... Jan 1 07:34:24.779653642 (640h5m24.650736677s) [upload=560864]: Next returned nil segment Jan 1 07:34:24.779669853 (640h5m24.650752898s) [upload=560864]: Waiting for error group managing segments... Jan 1 07:34:25.141584299 (640h5m25.012667344s) [noise=true , address=64.49.13.27:4960 , piece_id=DL7TWK35PNOJJDV3XSGTUZA4673I2ZNWEZPMFIRDGKRCADRNQQ4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.177436105 (640h5m25.04851915s) [noise=true , address=64.49.13.137:4493 , piece_id=W7OOHDHSHQVRBGNLCDISDDTJYYR2DOGIHTJPUARKBXGIAURGY4LA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.228023551 (640h5m25.099106596s) [noise=true , address=145.79.227.46:4634 , piece_id=ETPY2EKFZNF5FPQC4JVADBNEKNCCZZHPTJJSFUENOSUZAWHBDOUA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.241305873 (640h5m25.112388918s) [noise=true , address=145.79.227.6:4273 , piece_id=4SLG3X4PS7RLXSAKYL7W4UUO2TCUKUW3ZEGKHRZSBCZR47RSP7TQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263021061 (640h5m25.134104106s) [noise=true , address=145.79.227.35:4514 , piece_id=KBBEPMNCPOP4CQ36PTDC4CYSMYTIFMNP5ZK33ZNNIMZMJIQOEKGA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.263595455 (640h5m25.134678501s) [noise=true , address=145.79.227.43:4186 , piece_id=DXC3LKW6VQDCBQU5JPZEZPAVW3MXO65ID7CBZWES2UIUMFZ5UV3Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.296330898 (640h5m25.167413933s) [noise=true , address=145.79.227.5:4237 , piece_id=WRUF5X2JJY554AYUQMTK756FNVODMI7XT5B3W3HDSXZ77ZSW42PA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.374488530 (640h5m25.245571575s) [noise=true , address=145.79.227.34:4916 , piece_id=NLB5V4YYAPK3DSPAOPFKRAIAXUFDPJTWFRJ7XU6HB2LVKRREGL4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.375559300 (640h5m25.246642335s) [noise=true , address=77.90.31.19:4854 , piece_id=FIH57FGJOXPTTYQOEKBHYTMGQ72UERFIZCCZMYWQI3BEN4SQUA7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.491383705 (640h5m25.36246674s) [noise=true , address=77.90.31.13:4657 , piece_id=PA64O6J5X24R3X3YXU7Y6KL2WNBVKGXVPDIQRGXWXW4SETQSBE3A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:25.654523316 (640h5m25.525606351s) [noise=true , address=45.91.116.5:4325 , piece_id=X2Y6VPR32ARH3QIAUNWSKCS7HJIAQ24FK2TEJHVITPZCPUAJKELQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.108445006 (640h5m25.979528051s) [noise=true , address=145.79.227.49:4357 , piece_id=Y76L7HQURYDM7LZRWY3MCKGK4ZFPMEVBWSDO2WMSTCJKZGPD6Z4A, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.115852751 (640h5m25.986935796s) [noise=true , address=193.149.164.143:4701 , piece_id=P22D4RKMQRG5KIC3BTU6ZTDYVSBUQP5WFNOTPEM4G7GKIS3RI3QQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232049845 (640h5m26.10313288s) [noise=true , address=145.79.227.30:4785 , piece_id=NX3Y2OK2HLAFUUI52WLIM2CCM4DI3QWBKEOGFX3CXGC5CYMT4BUQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: <nil> Jan 1 07:34:26.232069563 (640h5m26.103152598s) [seg_pos={0 0}, upload=560864]: Segment reached optimal threshold of 14 pieces. Jan 1 07:34:26.232280164 (640h5m26.103363199s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232287498 (640h5m26.103370533s) [noise=true , address=145.79.227.33:4881 , piece_id=SE2VWLU62KDDISGUZ5ZEQTWSDHB56GHCTGGPS2ELHR7TONCXEEOA, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232483101 (640h5m26.103566136s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232501596 (640h5m26.103584641s) [noise=true , address=45.91.116.6:4003 , piece_id=WOQ3VNJ3CE2CBJQJMGDKTDGWPF4LT4CYNPGSWGBKAUAXXZFE3W7Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232516805 (640h5m26.10359984s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232523277 (640h5m26.103606312s) [noise=true , address=145.79.227.40:4058 , piece_id=7LE2VO42G4XDJUSKE2DBSKINWIAQ7U6TJKUAAF4GYMG3RNQFOO2Q, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232688692 (640h5m26.103771727s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH): protocol: expected piece hash; context canceled; EOF Jan 1 07:34:26.232694934 (640h5m26.103777969s) [noise=true , address=145.79.227.44:4218 , piece_id=VD6WI7B5GU3PRYHOARMKHUWMXRAE4X4J3GQAQCR2D5E76QCZSJYQ, seg_pos={0 0}, upload=560864]: Long tail cancellation due to optimal threshold Jan 1 07:34:26.232749067 (640h5m26.103832102s) [seg_pos={0 0}, upload=560864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:34:26.232844639 (640h5m26.103927684s) [upload=560864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:34:26.232850259 (640h5m26.103933305s) [upload=560864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:34:26.527641423 (640h5m26.398724458s) [upload=560864]: Done uploading segments... Jan 1 07:34:26.528300537 (640h5m26.399383573s) [upload=560864]: Upload finished. err: <nil> Duration: 59.335966939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X...] duration: 6.572087902s trace: https://jaeger.dev.storj.io/trace/689b57d373ec8394 FAIL (2026-01-01 07:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 07:33:13.155803378 (640h4m13.026886423s) [upload=560848]: Starting upload Jan 1 07:33:13.155867761 (640h4m13.026950796s) [upload=560848]: Done starting upload Jan 1 07:33:13.156062643 (640h4m13.027145678s) [upload=560848]: Uploading segments... Jan 1 07:33:13.156098731 (640h4m13.027181776s) [upload=560848]: Got next segment. Inline: false Jan 1 07:33:13.156140330 (640h4m13.027223365s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:33:13.156147163 (640h4m13.027230198s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:33:13.402949229 (640h4m13.274032264s) [seg_pos={0 0}, upload=560848]: Begin upload segment... Jan 1 07:33:13.403011137 (640h4m13.274094172s) [seg_pos={0 0}, upload=560848]: Done begin upload segment. Jan 1 07:33:13.403038499 (640h4m13.274121534s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403079968 (640h4m13.274163003s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403729014 (640h4m13.274812049s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404175845 (640h4m13.275258891s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404251229 (640h4m13.275334274s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404576989 (640h4m13.275660034s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404701476 (640h4m13.275784511s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405009122 (640h4m13.276092157s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405145422 (640h4m13.276228467s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405455401 (640h4m13.276538436s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405541526 (640h4m13.276624571s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.406988601 (640h4m13.278071636s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.407684506 (640h4m13.278767551s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408518093 (640h4m13.279601158s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408754133 (640h4m13.279837178s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.409230901 (640h4m13.280313936s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:15.111369780 (640h4m14.982452815s) [upload=560848]: Next returned nil segment Jan 1 07:33:15.111382736 (640h4m14.982465771s) [upload=560848]: Waiting for error group managing segments... Jan 1 07:33:15.398022337 (640h4m15.269105372s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.398545173 (640h4m15.269628208s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.404053602 (640h4m15.275136637s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418379150 (640h4m15.289462185s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418580073 (640h4m15.289663108s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420263598 (640h4m15.291346643s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420802935 (640h4m15.29188597s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.481391975 (640h4m15.35247502s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.563869022 (640h4m15.434952067s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.577943001 (640h4m15.449026046s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.940947854 (640h4m15.812030889s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.070209966 (640h4m16.941293001s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117283972 (640h4m16.988367007s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117315671 (640h4m16.988398706s) [seg_pos={0 0}, upload=560848]: Segment reached optimal threshold of 13 pieces. Jan 1 07:33:17.117523647 (640h4m16.988606683s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117529830 (640h4m16.988612865s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.117747114 (640h4m16.988830149s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117752474 (640h4m16.988835509s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118088173 (640h4m16.989171208s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.118105736 (640h4m16.989188771s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118152325 (640h4m16.98923536s) [seg_pos={0 0}, upload=560848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:33:17.118253227 (640h4m16.989336272s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:33:17.118258518 (640h4m16.989341553s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:33:17.406273468 (640h4m17.277356503s) [upload=560848]: Done uploading segments... Jan 1 07:33:17.406370764 (640h4m17.277453809s) [upload=560848]: Upload finished. err: <nil> Duration: 6.572107389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X..] duration: 6.2498054s trace: https://jaeger.dev.storj.io/trace/689b57d373ec8394 FAIL (2026-01-01 07:34:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 07:33:13.155803378 (640h4m13.026886423s) [upload=560848]: Starting upload Jan 1 07:33:13.155867761 (640h4m13.026950796s) [upload=560848]: Done starting upload Jan 1 07:33:13.156062643 (640h4m13.027145678s) [upload=560848]: Uploading segments... Jan 1 07:33:13.156098731 (640h4m13.027181776s) [upload=560848]: Got next segment. Inline: false Jan 1 07:33:13.156140330 (640h4m13.027223365s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:33:13.156147163 (640h4m13.027230198s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:33:13.402949229 (640h4m13.274032264s) [seg_pos={0 0}, upload=560848]: Begin upload segment... Jan 1 07:33:13.403011137 (640h4m13.274094172s) [seg_pos={0 0}, upload=560848]: Done begin upload segment. Jan 1 07:33:13.403038499 (640h4m13.274121534s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403079968 (640h4m13.274163003s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403729014 (640h4m13.274812049s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404175845 (640h4m13.275258891s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404251229 (640h4m13.275334274s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404576989 (640h4m13.275660034s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404701476 (640h4m13.275784511s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405009122 (640h4m13.276092157s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405145422 (640h4m13.276228467s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405455401 (640h4m13.276538436s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405541526 (640h4m13.276624571s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.406988601 (640h4m13.278071636s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.407684506 (640h4m13.278767551s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408518093 (640h4m13.279601158s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408754133 (640h4m13.279837178s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.409230901 (640h4m13.280313936s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:15.111369780 (640h4m14.982452815s) [upload=560848]: Next returned nil segment Jan 1 07:33:15.111382736 (640h4m14.982465771s) [upload=560848]: Waiting for error group managing segments... Jan 1 07:33:15.398022337 (640h4m15.269105372s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.398545173 (640h4m15.269628208s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.404053602 (640h4m15.275136637s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418379150 (640h4m15.289462185s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418580073 (640h4m15.289663108s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420263598 (640h4m15.291346643s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420802935 (640h4m15.29188597s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.481391975 (640h4m15.35247502s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.563869022 (640h4m15.434952067s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.577943001 (640h4m15.449026046s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.940947854 (640h4m15.812030889s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.070209966 (640h4m16.941293001s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117283972 (640h4m16.988367007s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117315671 (640h4m16.988398706s) [seg_pos={0 0}, upload=560848]: Segment reached optimal threshold of 13 pieces. Jan 1 07:33:17.117523647 (640h4m16.988606683s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117529830 (640h4m16.988612865s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.117747114 (640h4m16.988830149s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117752474 (640h4m16.988835509s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118088173 (640h4m16.989171208s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.118105736 (640h4m16.989188771s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118152325 (640h4m16.98923536s) [seg_pos={0 0}, upload=560848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:33:17.118253227 (640h4m16.989336272s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:33:17.118258518 (640h4m16.989341553s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:33:17.406273468 (640h4m17.277356503s) [upload=560848]: Done uploading segments... Jan 1 07:33:17.406370764 (640h4m17.277453809s) [upload=560848]: Upload finished. err: <nil> Duration: 6.249829898s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXX.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXXX] duration: 5.001544768s trace: https://jaeger.dev.storj.io/trace/3b17a466b8505b5c FAIL (2026-01-01 07:34:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001570868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..XX............X...........X.....................X.....X..X] duration: 1m0.003439823s trace: https://jaeger.dev.storj.io/trace/52aedd6d1348aa0 FAIL (2026-01-01 07:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (640h4m9.697109385s) [noise=true , address=145.79.227.44:4200 , piece_id=Z42XNCOEL5ZM3JCIBKPUJB5Y5XMTXNSZ32NOTUC3F2IW46ZXBZFQ, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.826119668 (640h4m9.697202713s) [noise=true , address=77.90.31.11:4582 , piece_id=W6O3LQNGX5HZZB5VPFXQRGTNOW2GFQWCHT6PTBVJ3HAVPWWOG6SA, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.826437042 (640h4m9.697520087s) [noise=true , address=145.79.227.11:4762 , piece_id=VZC5EDIUU45PP6SXJYLTNF2ULZCIONOUK37NDFTHWIJ3VQOP7A2Q, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.826482748 (640h4m9.697565794s) [noise=true , address=64.49.13.140:4506 , piece_id=6H4CCQRDOBDRODCDWYGULVTHSCBETGW3JKCMVXIGGXNDF53RMSQQ, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.826791476 (640h4m9.697874522s) [noise=true , address=193.149.164.143:4695 , piece_id=V57VGKQJLEQ5BRWS3WQHXSNF452Q5E2CKOOIKYKDIIJKF56LCMDA, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.826916374 (640h4m9.697999419s) [noise=true , address=145.79.227.33:4878 , piece_id=N376TBYDPK57HA5H7FDJFYHMSBA5R47FMVLTY5SI6W3TST3GOSUA, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.827143487 (640h4m9.698226532s) [noise=true , address=77.90.31.16:4755 , piece_id=PF5U4RZ5345EP2QKLC7KHLTWFSYO5555MEDEVDIDSXPTHKRFPGGA, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.827288182 (640h4m9.698371227s) [noise=true , address=77.90.31.21:4905 , piece_id=QUJ3IIQYJK7AX55EKZAZUY7EO36LBMFSNUB2K3ZP2SAA5KYM2HKQ, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.827461572 (640h4m9.698544617s) [noise=true , address=77.90.31.13:4635 , piece_id=L6EOVWENSPM3XWRQDFOPA5SAZVRHGNRY2MJ3Q7SLKEOY6RIYARNA, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.827839592 (640h4m9.698922637s) [noise=true , address=64.49.13.169:4705 , piece_id=UDNYBZIK3KYQVJKVUWX4WAHGDXIKUUI5BJQ23ZRLAIOSSS7B2PQQ, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.827858908 (640h4m9.698941953s) [noise=true , address=145.79.227.30:4784 , piece_id=HT6Z3BVHPDPOH6KC7NQLITWAKM6GTOVZ55SDIAFY3BYUOU6KZWAQ, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.828223874 (640h4m9.699306919s) [noise=true , address=145.79.227.29:4976 , piece_id=G3SRVVZPIB5ZICHAU7N6B5CDHAVFBA4GYOOEZJNBZ7I4XD5CDHKQ, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.828364211 (640h4m9.699447246s) [noise=true , address=77.90.31.15:4702 , piece_id=IOWOYGUP4HTQIGJF3IGXQUYTUG6UDSCJ7C5JTAB6R42BGL767RJA, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.828562469 (640h4m9.699645514s) [noise=true , address=64.49.13.170:4717 , piece_id=LVMASFQUWYJZHXIT6QSCXIHWL675UJPH5GHLHOMDK7ES3RWZHJEA, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:09.828863031 (640h4m9.699946076s) [noise=true , address=145.79.227.35:4506 , piece_id=OGSAGDAMH6O4Z744T75UTGYUXN3TYQ4GB3DIQKXGLKOXUIR2IPNQ, seg_pos={0 0}, upload=560847]: Uploading piece... Jan 1 07:33:11.274563126 (640h4m11.145646161s) [upload=560847]: Next returned nil segment Jan 1 07:33:11.274579718 (640h4m11.145662763s) [upload=560847]: Waiting for error group managing segments... Jan 1 07:33:11.645237231 (640h4m11.516320336s) [noise=true , address=64.49.13.170:4717 , piece_id=LVMASFQUWYJZHXIT6QSCXIHWL675UJPH5GHLHOMDK7ES3RWZHJEA, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:11.686508457 (640h4m11.557591492s) [noise=true , address=145.79.227.11:4762 , piece_id=VZC5EDIUU45PP6SXJYLTNF2ULZCIONOUK37NDFTHWIJ3VQOP7A2Q, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:11.687912982 (640h4m11.558996027s) [noise=true , address=145.79.227.43:4180 , piece_id=MYBMXGDTVZPM2B3VQRN7CNX7WGSKUPGMBXBDHC2O7PSY4QZLGN7A, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:11.713433306 (640h4m11.584516341s) [noise=true , address=145.79.227.29:4976 , piece_id=G3SRVVZPIB5ZICHAU7N6B5CDHAVFBA4GYOOEZJNBZ7I4XD5CDHKQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:11.828082294 (640h4m11.699165329s) [noise=true , address=64.49.13.140:4506 , piece_id=6H4CCQRDOBDRODCDWYGULVTHSCBETGW3JKCMVXIGGXNDF53RMSQQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:11.977857767 (640h4m11.848940812s) [noise=true , address=77.90.31.13:4635 , piece_id=L6EOVWENSPM3XWRQDFOPA5SAZVRHGNRY2MJ3Q7SLKEOY6RIYARNA, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:11.995494941 (640h4m11.866577986s) [noise=true , address=77.90.31.11:4582 , piece_id=W6O3LQNGX5HZZB5VPFXQRGTNOW2GFQWCHT6PTBVJ3HAVPWWOG6SA, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.158796639 (640h4m12.029879684s) [noise=true , address=77.90.31.16:4755 , piece_id=PF5U4RZ5345EP2QKLC7KHLTWFSYO5555MEDEVDIDSXPTHKRFPGGA, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.527718908 (640h4m12.398801943s) [noise=true , address=77.90.31.12:4618 , piece_id=F2LQZWN542J4LYIXAUYVPN27HJWSD4I6ZE3SNC6JAY7ZNVQHZ5KQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.536971537 (640h4m12.408054582s) [noise=true , address=145.79.227.44:4200 , piece_id=Z42XNCOEL5ZM3JCIBKPUJB5Y5XMTXNSZ32NOTUC3F2IW46ZXBZFQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.560192663 (640h4m12.431275708s) [noise=true , address=145.79.227.30:4784 , piece_id=HT6Z3BVHPDPOH6KC7NQLITWAKM6GTOVZ55SDIAFY3BYUOU6KZWAQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.613196199 (640h4m12.484279234s) [noise=true , address=145.79.227.33:4878 , piece_id=N376TBYDPK57HA5H7FDJFYHMSBA5R47FMVLTY5SI6W3TST3GOSUA, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.712863129 (640h4m12.583946174s) [noise=true , address=145.79.227.35:4506 , piece_id=OGSAGDAMH6O4Z744T75UTGYUXN3TYQ4GB3DIQKXGLKOXUIR2IPNQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.752146971 (640h4m12.623230006s) [noise=true , address=193.149.164.143:4695 , piece_id=V57VGKQJLEQ5BRWS3WQHXSNF452Q5E2CKOOIKYKDIIJKF56LCMDA, seg_pos={0 0}, upload=560847]: Done uploading piece. err: <nil> Jan 1 07:33:12.752166848 (640h4m12.623249883s) [seg_pos={0 0}, upload=560847]: Segment reached optimal threshold of 14 pieces. Jan 1 07:33:12.752333085 (640h4m12.62341612s) [noise=true , address=77.90.31.15:4702 , piece_id=IOWOYGUP4HTQIGJF3IGXQUYTUG6UDSCJ7C5JTAB6R42BGL767RJA, seg_pos={0 0}, upload=560847]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:35536->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:35536->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35536->77.90.31.15:4702: use of closed network connection Jan 1 07:33:12.752339276 (640h4m12.623422321s) [noise=true , address=77.90.31.15:4702 , piece_id=IOWOYGUP4HTQIGJF3IGXQUYTUG6UDSCJ7C5JTAB6R42BGL767RJA, seg_pos={0 0}, upload=560847]: Long tail cancellation due to optimal threshold Jan 1 07:33:12.752347863 (640h4m12.623430898s) [noise=true , address=77.90.31.21:4905 , piece_id=QUJ3IIQYJK7AX55EKZAZUY7EO36LBMFSNUB2K3ZP2SAA5KYM2HKQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:12.752356950 (640h4m12.623439995s) [noise=true , address=77.90.31.21:4905 , piece_id=QUJ3IIQYJK7AX55EKZAZUY7EO36LBMFSNUB2K3ZP2SAA5KYM2HKQ, seg_pos={0 0}, upload=560847]: Long tail cancellation due to optimal threshold Jan 1 07:33:12.752559456 (640h4m12.623642491s) [noise=true , address=145.79.227.49:4372 , piece_id=RJ7OV366M75SRO6AWGDTMT5NCO34HSGJ5SYSBYPFDL222SNT2XKQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:12.752566700 (640h4m12.623649735s) [noise=true , address=145.79.227.49:4372 , piece_id=RJ7OV366M75SRO6AWGDTMT5NCO34HSGJ5SYSBYPFDL222SNT2XKQ, seg_pos={0 0}, upload=560847]: Long tail cancellation due to optimal threshold Jan 1 07:33:12.752606095 (640h4m12.62368913s) [noise=true , address=64.49.13.169:4705 , piece_id=UDNYBZIK3KYQVJKVUWX4WAHGDXIKUUI5BJQ23ZRLAIOSSS7B2PQQ, seg_pos={0 0}, upload=560847]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51732->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51732->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51732->64.49.13.169:4705: use of closed network connection Jan 1 07:33:12.752614981 (640h4m12.623698026s) [noise=true , address=64.49.13.169:4705 , piece_id=UDNYBZIK3KYQVJKVUWX4WAHGDXIKUUI5BJQ23ZRLAIOSSS7B2PQQ, seg_pos={0 0}, upload=560847]: Long tail cancellation due to optimal threshold Jan 1 07:33:12.752631263 (640h4m12.623714308s) [seg_pos={0 0}, upload=560847]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:33:12.752698331 (640h4m12.623781366s) [upload=560847]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:33:12.752703039 (640h4m12.623786074s) [upload=560847]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:33:13.039106280 (640h4m12.910189315s) [upload=560847]: Done uploading segments... Jan 1 07:33:13.039131018 (640h4m12.910214063s) [upload=560847]: Upload finished. err: <nil> Duration: 1m0.003467364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XX........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X.] duration: 5.9426823s trace: https://jaeger.dev.storj.io/trace/689b57d373ec8394 FAIL (2026-01-01 07:33:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 07:33:13.155803378 (640h4m13.026886423s) [upload=560848]: Starting upload Jan 1 07:33:13.155867761 (640h4m13.026950796s) [upload=560848]: Done starting upload Jan 1 07:33:13.156062643 (640h4m13.027145678s) [upload=560848]: Uploading segments... Jan 1 07:33:13.156098731 (640h4m13.027181776s) [upload=560848]: Got next segment. Inline: false Jan 1 07:33:13.156140330 (640h4m13.027223365s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:33:13.156147163 (640h4m13.027230198s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:33:13.402949229 (640h4m13.274032264s) [seg_pos={0 0}, upload=560848]: Begin upload segment... Jan 1 07:33:13.403011137 (640h4m13.274094172s) [seg_pos={0 0}, upload=560848]: Done begin upload segment. Jan 1 07:33:13.403038499 (640h4m13.274121534s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403079968 (640h4m13.274163003s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403729014 (640h4m13.274812049s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404175845 (640h4m13.275258891s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404251229 (640h4m13.275334274s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404576989 (640h4m13.275660034s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404701476 (640h4m13.275784511s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405009122 (640h4m13.276092157s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405145422 (640h4m13.276228467s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405455401 (640h4m13.276538436s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405541526 (640h4m13.276624571s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.406988601 (640h4m13.278071636s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.407684506 (640h4m13.278767551s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408518093 (640h4m13.279601158s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408754133 (640h4m13.279837178s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.409230901 (640h4m13.280313936s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:15.111369780 (640h4m14.982452815s) [upload=560848]: Next returned nil segment Jan 1 07:33:15.111382736 (640h4m14.982465771s) [upload=560848]: Waiting for error group managing segments... Jan 1 07:33:15.398022337 (640h4m15.269105372s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.398545173 (640h4m15.269628208s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.404053602 (640h4m15.275136637s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418379150 (640h4m15.289462185s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418580073 (640h4m15.289663108s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420263598 (640h4m15.291346643s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420802935 (640h4m15.29188597s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.481391975 (640h4m15.35247502s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.563869022 (640h4m15.434952067s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.577943001 (640h4m15.449026046s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.940947854 (640h4m15.812030889s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.070209966 (640h4m16.941293001s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117283972 (640h4m16.988367007s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117315671 (640h4m16.988398706s) [seg_pos={0 0}, upload=560848]: Segment reached optimal threshold of 13 pieces. Jan 1 07:33:17.117523647 (640h4m16.988606683s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117529830 (640h4m16.988612865s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.117747114 (640h4m16.988830149s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117752474 (640h4m16.988835509s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118088173 (640h4m16.989171208s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.118105736 (640h4m16.989188771s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118152325 (640h4m16.98923536s) [seg_pos={0 0}, upload=560848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:33:17.118253227 (640h4m16.989336272s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:33:17.118258518 (640h4m16.989341553s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:33:17.406273468 (640h4m17.277356503s) [upload=560848]: Done uploading segments... Jan 1 07:33:17.406370764 (640h4m17.277453809s) [upload=560848]: Upload finished. err: <nil> Duration: 5.942696085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.XX........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.X] duration: 30.002879302s trace: https://jaeger.dev.storj.io/trace/689b57d373ec8394 FAIL (2026-01-01 07:33:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 07:33:13.155803378 (640h4m13.026886423s) [upload=560848]: Starting upload Jan 1 07:33:13.155867761 (640h4m13.026950796s) [upload=560848]: Done starting upload Jan 1 07:33:13.156062643 (640h4m13.027145678s) [upload=560848]: Uploading segments... Jan 1 07:33:13.156098731 (640h4m13.027181776s) [upload=560848]: Got next segment. Inline: false Jan 1 07:33:13.156140330 (640h4m13.027223365s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:33:13.156147163 (640h4m13.027230198s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:33:13.402949229 (640h4m13.274032264s) [seg_pos={0 0}, upload=560848]: Begin upload segment... Jan 1 07:33:13.403011137 (640h4m13.274094172s) [seg_pos={0 0}, upload=560848]: Done begin upload segment. Jan 1 07:33:13.403038499 (640h4m13.274121534s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403079968 (640h4m13.274163003s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.403729014 (640h4m13.274812049s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404175845 (640h4m13.275258891s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404251229 (640h4m13.275334274s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404576989 (640h4m13.275660034s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.404701476 (640h4m13.275784511s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405009122 (640h4m13.276092157s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405145422 (640h4m13.276228467s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405455401 (640h4m13.276538436s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.405541526 (640h4m13.276624571s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.406988601 (640h4m13.278071636s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.407684506 (640h4m13.278767551s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408518093 (640h4m13.279601158s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.408754133 (640h4m13.279837178s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:13.409230901 (640h4m13.280313936s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Uploading piece... Jan 1 07:33:15.111369780 (640h4m14.982452815s) [upload=560848]: Next returned nil segment Jan 1 07:33:15.111382736 (640h4m14.982465771s) [upload=560848]: Waiting for error group managing segments... Jan 1 07:33:15.398022337 (640h4m15.269105372s) [noise=true , address=145.79.227.26:4653 , piece_id=4YZJKFMLY56YWJ3CDIUEQLVP7ZIDUPH6IFIQHCUP5KMURL7NT2OA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.398545173 (640h4m15.269628208s) [noise=true , address=145.79.227.24:4581 , piece_id=2MEGBIA72QGXPKNA6T74GLCTLHSRRTWS4VLLSUAZP5VYL4YPJ3AA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.404053602 (640h4m15.275136637s) [noise=true , address=145.79.227.27:4689 , piece_id=57BLYHH2WVAFWIVD45NVRQKIYLXXYUJ3KIQPJSLQWSU7THNHGPNQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418379150 (640h4m15.289462185s) [noise=true , address=145.79.227.37:4565 , piece_id=OTUICFUVMKSEBK6R7VF66IKFNK5WBVBUGTK5THIK7MJ6UAV7DR3Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.418580073 (640h4m15.289663108s) [noise=true , address=145.79.227.21:4474 , piece_id=2I4YI7JS2XZ2742OVXMBPX6VUZVL6QSCORATATZ6GSAWMEFUQT6Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420263598 (640h4m15.291346643s) [noise=true , address=145.79.227.19:4077 , piece_id=56ZFES2IAEWSQMDBPDOMKOWUWGX6CQSGXZ5CC6K3IORYYLHOQLUA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.420802935 (640h4m15.29188597s) [noise=true , address=145.79.227.6:4257 , piece_id=RDWPCWQPV5TZLATQMUGDTG3JHDFETBLBXK6MWGRZFUDSXUXLY3LA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.481391975 (640h4m15.35247502s) [noise=true , address=145.79.227.48:4303 , piece_id=JABNUBOMXGUQZFWXDUTE5U3HHNDWURRA5MPBHHOFQFHTWZWWA2TQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.563869022 (640h4m15.434952067s) [noise=true , address=145.79.227.50:4375 , piece_id=HQSW3IP2CJS477MWBIJ257VVHZ3RXXT4QD3MZI6YMI6E2FDGFRLA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.577943001 (640h4m15.449026046s) [noise=true , address=77.90.31.17:4770 , piece_id=B5VYBBYWOND7SCSV6COKI4U4X6MUFVUIMAKKJAHOVBARL2KRJNJA, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:15.940947854 (640h4m15.812030889s) [noise=true , address=45.91.116.4:4939 , piece_id=I6R6CKTJCB3OK7VWNR4TVACHDZCTBV5O645U6LCNJRDE4URQTWVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.070209966 (640h4m16.941293001s) [noise=true , address=77.90.31.18:4805 , piece_id=65XVR2J2R35UF23DASOD2S5G52RWFMRWNLKBBAMGYFZQV5D2QFVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117283972 (640h4m16.988367007s) [noise=true , address=77.90.31.10:4518 , piece_id=QWXP6MT736EIBNW7O2YR7ZXSKLXWA6DJWXNS5HKS2DYKR4TANIVQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: <nil> Jan 1 07:33:17.117315671 (640h4m16.988398706s) [seg_pos={0 0}, upload=560848]: Segment reached optimal threshold of 13 pieces. Jan 1 07:33:17.117523647 (640h4m16.988606683s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117529830 (640h4m16.988612865s) [noise=true , address=145.79.227.33:4871 , piece_id=EEZCHP5QPTKE7EPQC2NN2DKXD2CGDC3QE7Q5JZ4AJDVJRRGZZURQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.117747114 (640h4m16.988830149s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.117752474 (640h4m16.988835509s) [noise=true , address=77.90.31.19:4841 , piece_id=NZ3NI2PO3FPPZZUVMNEDCB4IVGXQDQSRHWROQH7FSLF2HW56TY2Q, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118088173 (640h4m16.989171208s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 07:33:17.118105736 (640h4m16.989188771s) [noise=true , address=145.79.227.49:4338 , piece_id=3B62N5G35S6M5RPWJWXIFR4ERA7FBPPLZX7ZAQTYFS34XK6L4HAQ, seg_pos={0 0}, upload=560848]: Long tail cancellation due to optimal threshold Jan 1 07:33:17.118152325 (640h4m16.98923536s) [seg_pos={0 0}, upload=560848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:33:17.118253227 (640h4m16.989336272s) [upload=560848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:33:17.118258518 (640h4m16.989341553s) [upload=560848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:33:17.406273468 (640h4m17.277356503s) [upload=560848]: Done uploading segments... Jan 1 07:33:17.406370764 (640h4m17.277453809s) [upload=560848]: Upload finished. err: <nil> Duration: 30.002901413s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXX.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XXX] duration: 5.001239344s trace: https://jaeger.dev.storj.io/trace/1233d730b5e88c12 FAIL (2026-01-01 07:33:42 UTC) orbiter warning: context deadline exceeded Duration: 5.001262408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX............X...........X.....................X.....X..] duration: 16.862296431s trace: https://jaeger.dev.storj.io/trace/70a1e85d41a2cf70 FAIL (2026-01-01 07:33:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== MFIPHRKDSXY47IDNA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.864997229 (640h2m7.736080274s) [noise=true , address=145.79.227.17:4999 , piece_id=JZUMG5W7HHUB5MKPHGV5QOA4HI63MHQGGW43RBHXQGARNHUM7S4Q, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.865516548 (640h2m7.736599583s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866053551 (640h2m7.737136596s) [noise=true , address=145.79.227.52:4454 , piece_id=ZNRPDZXJW6NWPEQRZVFFECBQYJSTBTIAQZ43JRHB3KHT5RBWWTCA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866066876 (640h2m7.737149911s) [noise=true , address=145.79.227.40:4071 , piece_id=JSBG2ASARQZR7OAZUXNMWSHIHTEXFCB23DVHRWZFVNFBHPJEV4RQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866484080 (640h2m7.737567125s) [noise=true , address=77.90.31.14:4669 , piece_id=P5QENHQ7JQKVJYQLKQ4O3UA4S3MZKUTE6KRQF5KBC7VA226LX4TQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866554494 (640h2m7.73763754s) [noise=true , address=77.90.31.14:4696 , piece_id=3G3FLTWQY2LBJML7SLPVTUSBSZGACHPTCWFDIPGCOSIEOIPPSPTQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866878461 (640h2m7.737961506s) [noise=true , address=145.79.227.10:4412 , piece_id=G2HHTDKWAC723SXOTU77RSBXCJ5U2X7DV6MSEHCK7RXYQZKPBZ5A, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866927014 (640h2m7.738010049s) [noise=true , address=145.79.227.4:4192 , piece_id=5PMVDGHG54LQH7JQIBPZLXRZPCYMW2SFBNHKGVPCWLH4SKIDZWRA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.867276009 (640h2m7.738359054s) [noise=true , address=145.79.227.44:4211 , piece_id=BXZ3AUSEGUWVHGGBRL52UCB5253AEH6BCUBC7LLCSHZDZQEAPZLQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.867705596 (640h2m7.738788641s) [noise=true , address=145.79.227.49:4365 , piece_id=ZWEY36GNU5MRECJAYWI6VMTMPEAC3H6WEUATUUPSEYM3LKKY6FBA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868116399 (640h2m7.739199444s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868541698 (640h2m7.739624733s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868737772 (640h2m7.739820807s) [noise=true , address=45.91.116.6:4006 , piece_id=JT64JIMBSVQJKZLVYTJB6UG36IJYEJ5ZWQPVEONVGOKHTM27IJIQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.869182529 (640h2m7.740265564s) [noise=true , address=145.79.227.29:4949 , piece_id=HV4NWKEXPQZ2JMZ2DKWAC5F5E46GOMRJRV5JYXGY4H4F24GIGMOA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.869774086 (640h2m7.740857131s) [noise=true , address=64.49.13.26:4955 , piece_id=II6N2UZV5NRFLGNFVPMEW3Q3ZBSIMEKWXLNGQ34LCLUBGTTCTKQQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.870263738 (640h2m7.741346783s) [noise=true , address=77.90.31.21:4887 , piece_id=54SKDJSIGOY2S6CAGHZSHRFQP5POMZLTCO6H2CAHPFLQOFN7ZY2A, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.870718674 (640h2m7.741801719s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:09.286406211 (640h2m9.157489246s) [upload=560816]: Next returned nil segment Jan 1 07:31:09.286424576 (640h2m9.157507611s) [upload=560816]: Waiting for error group managing segments... Jan 1 07:31:09.646470543 (640h2m9.517553588s) [noise=true , address=64.49.13.26:4955 , piece_id=II6N2UZV5NRFLGNFVPMEW3Q3ZBSIMEKWXLNGQ34LCLUBGTTCTKQQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.729572180 (640h2m9.600655215s) [noise=true , address=145.79.227.29:4949 , piece_id=HV4NWKEXPQZ2JMZ2DKWAC5F5E46GOMRJRV5JYXGY4H4F24GIGMOA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.756915146 (640h2m9.627998191s) [noise=true , address=145.79.227.4:4192 , piece_id=5PMVDGHG54LQH7JQIBPZLXRZPCYMW2SFBNHKGVPCWLH4SKIDZWRA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.773518300 (640h2m9.644601335s) [noise=true , address=145.79.227.10:4412 , piece_id=G2HHTDKWAC723SXOTU77RSBXCJ5U2X7DV6MSEHCK7RXYQZKPBZ5A, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.813080993 (640h2m9.684164038s) [noise=true , address=145.79.227.17:4999 , piece_id=JZUMG5W7HHUB5MKPHGV5QOA4HI63MHQGGW43RBHXQGARNHUM7S4Q, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.898457802 (640h2m9.769540837s) [noise=true , address=77.90.31.14:4696 , piece_id=3G3FLTWQY2LBJML7SLPVTUSBSZGACHPTCWFDIPGCOSIEOIPPSPTQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.989265563 (640h2m9.860348608s) [noise=true , address=145.79.227.40:4071 , piece_id=JSBG2ASARQZR7OAZUXNMWSHIHTEXFCB23DVHRWZFVNFBHPJEV4RQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.004850749 (640h2m9.875933795s) [noise=true , address=77.90.31.14:4669 , piece_id=P5QENHQ7JQKVJYQLKQ4O3UA4S3MZKUTE6KRQF5KBC7VA226LX4TQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.022329011 (640h2m9.893412056s) [noise=true , address=145.79.227.52:4454 , piece_id=ZNRPDZXJW6NWPEQRZVFFECBQYJSTBTIAQZ43JRHB3KHT5RBWWTCA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.077129869 (640h2m9.948212914s) [noise=true , address=77.90.31.21:4887 , piece_id=54SKDJSIGOY2S6CAGHZSHRFQP5POMZLTCO6H2CAHPFLQOFN7ZY2A, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.116925085 (640h2m9.98800812s) [noise=true , address=45.91.116.6:4006 , piece_id=JT64JIMBSVQJKZLVYTJB6UG36IJYEJ5ZWQPVEONVGOKHTM27IJIQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.620867179 (640h2m10.491950214s) [noise=true , address=145.79.227.35:4494 , piece_id=VVD37LPMV523QCTFI2FR3YW3MGGY23VWEB6MFIPHRKDSXY47IDNA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.624754562 (640h2m10.495837607s) [noise=true , address=145.79.227.49:4365 , piece_id=ZWEY36GNU5MRECJAYWI6VMTMPEAC3H6WEUATUUPSEYM3LKKY6FBA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.625559685 (640h2m10.49664273s) [noise=true , address=145.79.227.44:4211 , piece_id=BXZ3AUSEGUWVHGGBRL52UCB5253AEH6BCUBC7LLCSHZDZQEAPZLQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.625581687 (640h2m10.496664732s) [seg_pos={0 0}, upload=560816]: Segment reached optimal threshold of 14 pieces. Jan 1 07:31:10.625942523 (640h2m10.497025599s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.625951421 (640h2m10.497034456s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.625964065 (640h2m10.4970471s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.625973132 (640h2m10.497056177s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626108109 (640h2m10.497191144s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.626115994 (640h2m10.497199029s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626251302 (640h2m10.497334347s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PRN34y7cCWokJkxgu5Zk4eBjDPW5iddNefMKXvUkJUTAqkKve): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.626258255 (640h2m10.4973413s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626318860 (640h2m10.497401905s) [seg_pos={0 0}, upload=560816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:31:10.626397239 (640h2m10.497480284s) [upload=560816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:31:10.626402520 (640h2m10.497485565s) [upload=560816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:31:10.922618708 (640h2m10.793701743s) [upload=560816]: Done uploading segments... Jan 1 07:31:10.922646190 (640h2m10.793729225s) [upload=560816]: Upload finished. err: <nil> Duration: 16.862311338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX............X...........X.....................X.....X.] duration: 44.217306782s trace: https://jaeger.dev.storj.io/trace/70a1e85d41a2cf70 FAIL (2026-01-01 07:32:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== MFIPHRKDSXY47IDNA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.864997229 (640h2m7.736080274s) [noise=true , address=145.79.227.17:4999 , piece_id=JZUMG5W7HHUB5MKPHGV5QOA4HI63MHQGGW43RBHXQGARNHUM7S4Q, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.865516548 (640h2m7.736599583s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866053551 (640h2m7.737136596s) [noise=true , address=145.79.227.52:4454 , piece_id=ZNRPDZXJW6NWPEQRZVFFECBQYJSTBTIAQZ43JRHB3KHT5RBWWTCA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866066876 (640h2m7.737149911s) [noise=true , address=145.79.227.40:4071 , piece_id=JSBG2ASARQZR7OAZUXNMWSHIHTEXFCB23DVHRWZFVNFBHPJEV4RQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866484080 (640h2m7.737567125s) [noise=true , address=77.90.31.14:4669 , piece_id=P5QENHQ7JQKVJYQLKQ4O3UA4S3MZKUTE6KRQF5KBC7VA226LX4TQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866554494 (640h2m7.73763754s) [noise=true , address=77.90.31.14:4696 , piece_id=3G3FLTWQY2LBJML7SLPVTUSBSZGACHPTCWFDIPGCOSIEOIPPSPTQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866878461 (640h2m7.737961506s) [noise=true , address=145.79.227.10:4412 , piece_id=G2HHTDKWAC723SXOTU77RSBXCJ5U2X7DV6MSEHCK7RXYQZKPBZ5A, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866927014 (640h2m7.738010049s) [noise=true , address=145.79.227.4:4192 , piece_id=5PMVDGHG54LQH7JQIBPZLXRZPCYMW2SFBNHKGVPCWLH4SKIDZWRA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.867276009 (640h2m7.738359054s) [noise=true , address=145.79.227.44:4211 , piece_id=BXZ3AUSEGUWVHGGBRL52UCB5253AEH6BCUBC7LLCSHZDZQEAPZLQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.867705596 (640h2m7.738788641s) [noise=true , address=145.79.227.49:4365 , piece_id=ZWEY36GNU5MRECJAYWI6VMTMPEAC3H6WEUATUUPSEYM3LKKY6FBA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868116399 (640h2m7.739199444s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868541698 (640h2m7.739624733s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868737772 (640h2m7.739820807s) [noise=true , address=45.91.116.6:4006 , piece_id=JT64JIMBSVQJKZLVYTJB6UG36IJYEJ5ZWQPVEONVGOKHTM27IJIQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.869182529 (640h2m7.740265564s) [noise=true , address=145.79.227.29:4949 , piece_id=HV4NWKEXPQZ2JMZ2DKWAC5F5E46GOMRJRV5JYXGY4H4F24GIGMOA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.869774086 (640h2m7.740857131s) [noise=true , address=64.49.13.26:4955 , piece_id=II6N2UZV5NRFLGNFVPMEW3Q3ZBSIMEKWXLNGQ34LCLUBGTTCTKQQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.870263738 (640h2m7.741346783s) [noise=true , address=77.90.31.21:4887 , piece_id=54SKDJSIGOY2S6CAGHZSHRFQP5POMZLTCO6H2CAHPFLQOFN7ZY2A, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.870718674 (640h2m7.741801719s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:09.286406211 (640h2m9.157489246s) [upload=560816]: Next returned nil segment Jan 1 07:31:09.286424576 (640h2m9.157507611s) [upload=560816]: Waiting for error group managing segments... Jan 1 07:31:09.646470543 (640h2m9.517553588s) [noise=true , address=64.49.13.26:4955 , piece_id=II6N2UZV5NRFLGNFVPMEW3Q3ZBSIMEKWXLNGQ34LCLUBGTTCTKQQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.729572180 (640h2m9.600655215s) [noise=true , address=145.79.227.29:4949 , piece_id=HV4NWKEXPQZ2JMZ2DKWAC5F5E46GOMRJRV5JYXGY4H4F24GIGMOA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.756915146 (640h2m9.627998191s) [noise=true , address=145.79.227.4:4192 , piece_id=5PMVDGHG54LQH7JQIBPZLXRZPCYMW2SFBNHKGVPCWLH4SKIDZWRA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.773518300 (640h2m9.644601335s) [noise=true , address=145.79.227.10:4412 , piece_id=G2HHTDKWAC723SXOTU77RSBXCJ5U2X7DV6MSEHCK7RXYQZKPBZ5A, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.813080993 (640h2m9.684164038s) [noise=true , address=145.79.227.17:4999 , piece_id=JZUMG5W7HHUB5MKPHGV5QOA4HI63MHQGGW43RBHXQGARNHUM7S4Q, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.898457802 (640h2m9.769540837s) [noise=true , address=77.90.31.14:4696 , piece_id=3G3FLTWQY2LBJML7SLPVTUSBSZGACHPTCWFDIPGCOSIEOIPPSPTQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.989265563 (640h2m9.860348608s) [noise=true , address=145.79.227.40:4071 , piece_id=JSBG2ASARQZR7OAZUXNMWSHIHTEXFCB23DVHRWZFVNFBHPJEV4RQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.004850749 (640h2m9.875933795s) [noise=true , address=77.90.31.14:4669 , piece_id=P5QENHQ7JQKVJYQLKQ4O3UA4S3MZKUTE6KRQF5KBC7VA226LX4TQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.022329011 (640h2m9.893412056s) [noise=true , address=145.79.227.52:4454 , piece_id=ZNRPDZXJW6NWPEQRZVFFECBQYJSTBTIAQZ43JRHB3KHT5RBWWTCA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.077129869 (640h2m9.948212914s) [noise=true , address=77.90.31.21:4887 , piece_id=54SKDJSIGOY2S6CAGHZSHRFQP5POMZLTCO6H2CAHPFLQOFN7ZY2A, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.116925085 (640h2m9.98800812s) [noise=true , address=45.91.116.6:4006 , piece_id=JT64JIMBSVQJKZLVYTJB6UG36IJYEJ5ZWQPVEONVGOKHTM27IJIQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.620867179 (640h2m10.491950214s) [noise=true , address=145.79.227.35:4494 , piece_id=VVD37LPMV523QCTFI2FR3YW3MGGY23VWEB6MFIPHRKDSXY47IDNA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.624754562 (640h2m10.495837607s) [noise=true , address=145.79.227.49:4365 , piece_id=ZWEY36GNU5MRECJAYWI6VMTMPEAC3H6WEUATUUPSEYM3LKKY6FBA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.625559685 (640h2m10.49664273s) [noise=true , address=145.79.227.44:4211 , piece_id=BXZ3AUSEGUWVHGGBRL52UCB5253AEH6BCUBC7LLCSHZDZQEAPZLQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.625581687 (640h2m10.496664732s) [seg_pos={0 0}, upload=560816]: Segment reached optimal threshold of 14 pieces. Jan 1 07:31:10.625942523 (640h2m10.497025599s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.625951421 (640h2m10.497034456s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.625964065 (640h2m10.4970471s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.625973132 (640h2m10.497056177s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626108109 (640h2m10.497191144s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.626115994 (640h2m10.497199029s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626251302 (640h2m10.497334347s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PRN34y7cCWokJkxgu5Zk4eBjDPW5iddNefMKXvUkJUTAqkKve): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.626258255 (640h2m10.4973413s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626318860 (640h2m10.497401905s) [seg_pos={0 0}, upload=560816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:31:10.626397239 (640h2m10.497480284s) [upload=560816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:31:10.626402520 (640h2m10.497485565s) [upload=560816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:31:10.922618708 (640h2m10.793701743s) [upload=560816]: Done uploading segments... Jan 1 07:31:10.922646190 (640h2m10.793729225s) [upload=560816]: Upload finished. err: <nil> Duration: 44.217348331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX.XX........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX.] duration: 5.730910767s trace: https://jaeger.dev.storj.io/trace/409747dc5088aec8 FAIL (2026-01-01 07:32:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: JCSQ6ISTVBVKATEP6AZ2F4R5VCYCAD6PYKVY3LZSUIO7UVUX4YSA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: JSAMISCN24MF3L4ZOTPVYF6Q74FAHCMDWCDIB45ST7733ZBJ3ADQ) context deadline exceeded ======LOGS====== Jan 1 07:32:12.189051932 (640h3m12.060134968s) [upload=560833]: Starting upload Jan 1 07:32:12.189104412 (640h3m12.060187457s) [upload=560833]: Done starting upload Jan 1 07:32:12.189293503 (640h3m12.060376548s) [upload=560833]: Uploading segments... Jan 1 07:32:12.189329802 (640h3m12.060412837s) [upload=560833]: Got next segment. Inline: false Jan 1 07:32:12.189356211 (640h3m12.060439246s) [upload=560833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:32:12.189361201 (640h3m12.060444237s) [upload=560833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:32:12.436934606 (640h3m12.308017651s) [seg_pos={0 0}, upload=560833]: Begin upload segment... Jan 1 07:32:12.436998558 (640h3m12.308081603s) [seg_pos={0 0}, upload=560833]: Done begin upload segment. Jan 1 07:32:12.437031159 (640h3m12.308114204s) [noise=true , address=45.91.116.8:4048 , piece_id=JCSQ6ISTVBVKATEP6AZ2F4R5VCYCAD6PYKVY3LZSUIO7UVUX4YSA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.437066015 (640h3m12.308149061s) [noise=true , address=145.79.227.42:4123 , piece_id=MJMLKUMZ37S5YHJB37EJ4DDFZ663GVHEEWVICUSVP3Y3B3GMO36A, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.437861160 (640h3m12.308944205s) [noise=true , address=145.79.227.12:4767 , piece_id=ZORAO6U4NSWPRKMMPCHLIPGAVHZR3R4TFJJSHDNZUOKWIBG6BCLQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438072763 (640h3m12.309155808s) [noise=true , address=77.90.31.21:4878 , piece_id=JSAMISCN24MF3L4ZOTPVYF6Q74FAHCMDWCDIB45ST7733ZBJ3ADQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438276150 (640h3m12.309359185s) [noise=true , address=145.79.227.16:4911 , piece_id=AKG5R5QY5T4CG5PY5KL2YJPL6DZCQW4KCC5F2VNO5UQB4NQDOMPA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438638080 (640h3m12.309721115s) [noise=true , address=145.79.227.28:4727 , piece_id=GXEFU4S4AD7NZ4HJW7DGCB7IFUUN7IXPCMW5EM5BG4III6ETFXEQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438661824 (640h3m12.309744869s) [noise=true , address=145.79.227.49:4338 , piece_id=BM5YJLW6ORBDFBJ5EY2YAK3G6C6NYVOIHTJZ66URITTTSQPNBL7A, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439039534 (640h3m12.310122569s) [noise=true , address=145.79.227.11:4731 , piece_id=CFTPKOK6EE2N5OE6Y3RSUHBAHIZHBPNEPQA4XN77L3AQFK6RAUZQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439162789 (640h3m12.310245834s) [noise=true , address=145.79.227.31:4798 , piece_id=EKUBYVZZAIM4TEMUFCMAXMLDKNE3UYEH6ZLOH3AR55UCJ34S3GXA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439441630 (640h3m12.310524665s) [noise=true , address=145.79.227.3:4151 , piece_id=KGIZ24OXLICAG3EHG2M6JXI4ZTJDUKNQLDIJNH2EC3P3VZQYYNBQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439701324 (640h3m12.310784369s) [noise=true , address=145.79.227.14:4840 , piece_id=2P7FYELBHQRSORTLOKMGH5IPMVCFEMB5LJ7RRGZC6PVIXSFJSWVA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439838074 (640h3m12.310921119s) [noise=true , address=145.79.227.36:4518 , piece_id=LVJ6HKF2TDZIBD7GFXJAZOJSPEJ3DIQW7YXCF4ZUWQHPY2HSJHSA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.440216415 (640h3m12.31129946s) [noise=true , address=145.79.227.9:4365 , piece_id=QE7PVBLHDAFDVFS6QFYSSGYMWD4ZKBCTL5ZCLAFQ6VA4XXHYI34A, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.440238857 (640h3m12.311321902s) [noise=true , address=77.90.31.18:4808 , piece_id=JJQ2WF2AKE7MIZ23SNVIZQ5GLZ5OEYKEOMARKHZUBWCE3AF4BAYA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.440648658 (640h3m12.311731693s) [noise=true , address=145.79.227.7:4295 , piece_id=E4SRB65EI5M7IE2GSVDBNDCMC2FPN5W5W6ZM3GL6WKD5GE3GCVVQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.441980273 (640h3m12.313063318s) [noise=true , address=45.91.116.10:4081 , piece_id=EUQ7ZENDA6RGKNZHYVD7GW6LL6EYXHLJS2DLGCWHBZW6HWXC5TXQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:14.136017597 (640h3m14.007100632s) [upload=560833]: Next returned nil segment Jan 1 07:32:14.136029900 (640h3m14.007112935s) [upload=560833]: Waiting for error group managing segments... Jan 1 07:32:14.416314021 (640h3m14.287397066s) [noise=true , address=145.79.227.9:4365 , piece_id=QE7PVBLHDAFDVFS6QFYSSGYMWD4ZKBCTL5ZCLAFQ6VA4XXHYI34A, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.426331966 (640h3m14.297415001s) [noise=true , address=145.79.227.14:4840 , piece_id=2P7FYELBHQRSORTLOKMGH5IPMVCFEMB5LJ7RRGZC6PVIXSFJSWVA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.433154728 (640h3m14.304237773s) [noise=true , address=145.79.227.16:4911 , piece_id=AKG5R5QY5T4CG5PY5KL2YJPL6DZCQW4KCC5F2VNO5UQB4NQDOMPA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.435950862 (640h3m14.307033897s) [noise=true , address=145.79.227.12:4767 , piece_id=ZORAO6U4NSWPRKMMPCHLIPGAVHZR3R4TFJJSHDNZUOKWIBG6BCLQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.436542879 (640h3m14.307625924s) [noise=true , address=145.79.227.36:4518 , piece_id=LVJ6HKF2TDZIBD7GFXJAZOJSPEJ3DIQW7YXCF4ZUWQHPY2HSJHSA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.438288764 (640h3m14.309371799s) [noise=true , address=145.79.227.28:4727 , piece_id=GXEFU4S4AD7NZ4HJW7DGCB7IFUUN7IXPCMW5EM5BG4III6ETFXEQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.438668457 (640h3m14.309751502s) [noise=true , address=145.79.227.11:4731 , piece_id=CFTPKOK6EE2N5OE6Y3RSUHBAHIZHBPNEPQA4XN77L3AQFK6RAUZQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.454806576 (640h3m14.325889621s) [noise=true , address=145.79.227.31:4798 , piece_id=EKUBYVZZAIM4TEMUFCMAXMLDKNE3UYEH6ZLOH3AR55UCJ34S3GXA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.459087387 (640h3m14.330170422s) [noise=true , address=145.79.227.42:4123 , piece_id=MJMLKUMZ37S5YHJB37EJ4DDFZ663GVHEEWVICUSVP3Y3B3GMO36A, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.462485769 (640h3m14.333568804s) [noise=true , address=145.79.227.7:4295 , piece_id=E4SRB65EI5M7IE2GSVDBNDCMC2FPN5W5W6ZM3GL6WKD5GE3GCVVQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.602432576 (640h3m14.473515621s) [noise=true , address=145.79.227.3:4151 , piece_id=KGIZ24OXLICAG3EHG2M6JXI4ZTJDUKNQLDIJNH2EC3P3VZQYYNBQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.719343992 (640h3m14.590427027s) [noise=true , address=77.90.31.21:4878 , piece_id=JSAMISCN24MF3L4ZOTPVYF6Q74FAHCMDWCDIB45ST7733ZBJ3ADQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:15.094974576 (640h3m14.966057621s) [noise=true , address=45.91.116.8:4048 , piece_id=JCSQ6ISTVBVKATEP6AZ2F4R5VCYCAD6PYKVY3LZSUIO7UVUX4YSA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:15.094994444 (640h3m14.966077499s) [seg_pos={0 0}, upload=560833]: Segment reached optimal threshold of 13 pieces. Jan 1 07:32:15.095188804 (640h3m14.966271839s) [noise=true , address=45.91.116.10:4081 , piece_id=EUQ7ZENDA6RGKNZHYVD7GW6LL6EYXHLJS2DLGCWHBZW6HWXC5TXQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Jan 1 07:32:15.095195517 (640h3m14.966278552s) [noise=true , address=45.91.116.10:4081 , piece_id=EUQ7ZENDA6RGKNZHYVD7GW6LL6EYXHLJS2DLGCWHBZW6HWXC5TXQ, seg_pos={0 0}, upload=560833]: Long tail cancellation due to optimal threshold Jan 1 07:32:15.095362334 (640h3m14.96644537s) [noise=true , address=145.79.227.49:4338 , piece_id=BM5YJLW6ORBDFBJ5EY2YAK3G6C6NYVOIHTJZ66URITTTSQPNBL7A, seg_pos={0 0}, upload=560833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 07:32:15.095368586 (640h3m14.966451621s) [noise=true , address=145.79.227.49:4338 , piece_id=BM5YJLW6ORBDFBJ5EY2YAK3G6C6NYVOIHTJZ66URITTTSQPNBL7A, seg_pos={0 0}, upload=560833]: Long tail cancellation due to optimal threshold Jan 1 07:32:15.095497662 (640h3m14.966580707s) [noise=true , address=77.90.31.18:4808 , piece_id=JJQ2WF2AKE7MIZ23SNVIZQ5GLZ5OEYKEOMARKHZUBWCE3AF4BAYA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:51612->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:51612->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51612->77.90.31.18:4808: use of closed network connection Jan 1 07:32:15.095518582 (640h3m14.966601617s) [noise=true , address=77.90.31.18:4808 , piece_id=JJQ2WF2AKE7MIZ23SNVIZQ5GLZ5OEYKEOMARKHZUBWCE3AF4BAYA, seg_pos={0 0}, upload=560833]: Long tail cancellation due to optimal threshold Jan 1 07:32:15.095555041 (640h3m14.966638086s) [seg_pos={0 0}, upload=560833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:32:15.095678126 (640h3m14.966761161s) [upload=560833]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:32:15.095684959 (640h3m14.966767994s) [upload=560833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:32:15.383970715 (640h3m15.25505375s) [upload=560833]: Done uploading segments... Jan 1 07:32:15.383989541 (640h3m15.255072586s) [upload=560833]: Upload finished. err: <nil> Duration: 5.730926456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX.XX........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.XX] duration: 30.002994627s trace: https://jaeger.dev.storj.io/trace/409747dc5088aec8 FAIL (2026-01-01 07:32:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: JCSQ6ISTVBVKATEP6AZ2F4R5VCYCAD6PYKVY3LZSUIO7UVUX4YSA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: JSAMISCN24MF3L4ZOTPVYF6Q74FAHCMDWCDIB45ST7733ZBJ3ADQ) context deadline exceeded ======LOGS====== Jan 1 07:32:12.189051932 (640h3m12.060134968s) [upload=560833]: Starting upload Jan 1 07:32:12.189104412 (640h3m12.060187457s) [upload=560833]: Done starting upload Jan 1 07:32:12.189293503 (640h3m12.060376548s) [upload=560833]: Uploading segments... Jan 1 07:32:12.189329802 (640h3m12.060412837s) [upload=560833]: Got next segment. Inline: false Jan 1 07:32:12.189356211 (640h3m12.060439246s) [upload=560833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:32:12.189361201 (640h3m12.060444237s) [upload=560833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:32:12.436934606 (640h3m12.308017651s) [seg_pos={0 0}, upload=560833]: Begin upload segment... Jan 1 07:32:12.436998558 (640h3m12.308081603s) [seg_pos={0 0}, upload=560833]: Done begin upload segment. Jan 1 07:32:12.437031159 (640h3m12.308114204s) [noise=true , address=45.91.116.8:4048 , piece_id=JCSQ6ISTVBVKATEP6AZ2F4R5VCYCAD6PYKVY3LZSUIO7UVUX4YSA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.437066015 (640h3m12.308149061s) [noise=true , address=145.79.227.42:4123 , piece_id=MJMLKUMZ37S5YHJB37EJ4DDFZ663GVHEEWVICUSVP3Y3B3GMO36A, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.437861160 (640h3m12.308944205s) [noise=true , address=145.79.227.12:4767 , piece_id=ZORAO6U4NSWPRKMMPCHLIPGAVHZR3R4TFJJSHDNZUOKWIBG6BCLQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438072763 (640h3m12.309155808s) [noise=true , address=77.90.31.21:4878 , piece_id=JSAMISCN24MF3L4ZOTPVYF6Q74FAHCMDWCDIB45ST7733ZBJ3ADQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438276150 (640h3m12.309359185s) [noise=true , address=145.79.227.16:4911 , piece_id=AKG5R5QY5T4CG5PY5KL2YJPL6DZCQW4KCC5F2VNO5UQB4NQDOMPA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438638080 (640h3m12.309721115s) [noise=true , address=145.79.227.28:4727 , piece_id=GXEFU4S4AD7NZ4HJW7DGCB7IFUUN7IXPCMW5EM5BG4III6ETFXEQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.438661824 (640h3m12.309744869s) [noise=true , address=145.79.227.49:4338 , piece_id=BM5YJLW6ORBDFBJ5EY2YAK3G6C6NYVOIHTJZ66URITTTSQPNBL7A, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439039534 (640h3m12.310122569s) [noise=true , address=145.79.227.11:4731 , piece_id=CFTPKOK6EE2N5OE6Y3RSUHBAHIZHBPNEPQA4XN77L3AQFK6RAUZQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439162789 (640h3m12.310245834s) [noise=true , address=145.79.227.31:4798 , piece_id=EKUBYVZZAIM4TEMUFCMAXMLDKNE3UYEH6ZLOH3AR55UCJ34S3GXA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439441630 (640h3m12.310524665s) [noise=true , address=145.79.227.3:4151 , piece_id=KGIZ24OXLICAG3EHG2M6JXI4ZTJDUKNQLDIJNH2EC3P3VZQYYNBQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439701324 (640h3m12.310784369s) [noise=true , address=145.79.227.14:4840 , piece_id=2P7FYELBHQRSORTLOKMGH5IPMVCFEMB5LJ7RRGZC6PVIXSFJSWVA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.439838074 (640h3m12.310921119s) [noise=true , address=145.79.227.36:4518 , piece_id=LVJ6HKF2TDZIBD7GFXJAZOJSPEJ3DIQW7YXCF4ZUWQHPY2HSJHSA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.440216415 (640h3m12.31129946s) [noise=true , address=145.79.227.9:4365 , piece_id=QE7PVBLHDAFDVFS6QFYSSGYMWD4ZKBCTL5ZCLAFQ6VA4XXHYI34A, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.440238857 (640h3m12.311321902s) [noise=true , address=77.90.31.18:4808 , piece_id=JJQ2WF2AKE7MIZ23SNVIZQ5GLZ5OEYKEOMARKHZUBWCE3AF4BAYA, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.440648658 (640h3m12.311731693s) [noise=true , address=145.79.227.7:4295 , piece_id=E4SRB65EI5M7IE2GSVDBNDCMC2FPN5W5W6ZM3GL6WKD5GE3GCVVQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:12.441980273 (640h3m12.313063318s) [noise=true , address=45.91.116.10:4081 , piece_id=EUQ7ZENDA6RGKNZHYVD7GW6LL6EYXHLJS2DLGCWHBZW6HWXC5TXQ, seg_pos={0 0}, upload=560833]: Uploading piece... Jan 1 07:32:14.136017597 (640h3m14.007100632s) [upload=560833]: Next returned nil segment Jan 1 07:32:14.136029900 (640h3m14.007112935s) [upload=560833]: Waiting for error group managing segments... Jan 1 07:32:14.416314021 (640h3m14.287397066s) [noise=true , address=145.79.227.9:4365 , piece_id=QE7PVBLHDAFDVFS6QFYSSGYMWD4ZKBCTL5ZCLAFQ6VA4XXHYI34A, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.426331966 (640h3m14.297415001s) [noise=true , address=145.79.227.14:4840 , piece_id=2P7FYELBHQRSORTLOKMGH5IPMVCFEMB5LJ7RRGZC6PVIXSFJSWVA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.433154728 (640h3m14.304237773s) [noise=true , address=145.79.227.16:4911 , piece_id=AKG5R5QY5T4CG5PY5KL2YJPL6DZCQW4KCC5F2VNO5UQB4NQDOMPA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.435950862 (640h3m14.307033897s) [noise=true , address=145.79.227.12:4767 , piece_id=ZORAO6U4NSWPRKMMPCHLIPGAVHZR3R4TFJJSHDNZUOKWIBG6BCLQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.436542879 (640h3m14.307625924s) [noise=true , address=145.79.227.36:4518 , piece_id=LVJ6HKF2TDZIBD7GFXJAZOJSPEJ3DIQW7YXCF4ZUWQHPY2HSJHSA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.438288764 (640h3m14.309371799s) [noise=true , address=145.79.227.28:4727 , piece_id=GXEFU4S4AD7NZ4HJW7DGCB7IFUUN7IXPCMW5EM5BG4III6ETFXEQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.438668457 (640h3m14.309751502s) [noise=true , address=145.79.227.11:4731 , piece_id=CFTPKOK6EE2N5OE6Y3RSUHBAHIZHBPNEPQA4XN77L3AQFK6RAUZQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.454806576 (640h3m14.325889621s) [noise=true , address=145.79.227.31:4798 , piece_id=EKUBYVZZAIM4TEMUFCMAXMLDKNE3UYEH6ZLOH3AR55UCJ34S3GXA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.459087387 (640h3m14.330170422s) [noise=true , address=145.79.227.42:4123 , piece_id=MJMLKUMZ37S5YHJB37EJ4DDFZ663GVHEEWVICUSVP3Y3B3GMO36A, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.462485769 (640h3m14.333568804s) [noise=true , address=145.79.227.7:4295 , piece_id=E4SRB65EI5M7IE2GSVDBNDCMC2FPN5W5W6ZM3GL6WKD5GE3GCVVQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.602432576 (640h3m14.473515621s) [noise=true , address=145.79.227.3:4151 , piece_id=KGIZ24OXLICAG3EHG2M6JXI4ZTJDUKNQLDIJNH2EC3P3VZQYYNBQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:14.719343992 (640h3m14.590427027s) [noise=true , address=77.90.31.21:4878 , piece_id=JSAMISCN24MF3L4ZOTPVYF6Q74FAHCMDWCDIB45ST7733ZBJ3ADQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:15.094974576 (640h3m14.966057621s) [noise=true , address=45.91.116.8:4048 , piece_id=JCSQ6ISTVBVKATEP6AZ2F4R5VCYCAD6PYKVY3LZSUIO7UVUX4YSA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: <nil> Jan 1 07:32:15.094994444 (640h3m14.966077499s) [seg_pos={0 0}, upload=560833]: Segment reached optimal threshold of 13 pieces. Jan 1 07:32:15.095188804 (640h3m14.966271839s) [noise=true , address=45.91.116.10:4081 , piece_id=EUQ7ZENDA6RGKNZHYVD7GW6LL6EYXHLJS2DLGCWHBZW6HWXC5TXQ, seg_pos={0 0}, upload=560833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Jan 1 07:32:15.095195517 (640h3m14.966278552s) [noise=true , address=45.91.116.10:4081 , piece_id=EUQ7ZENDA6RGKNZHYVD7GW6LL6EYXHLJS2DLGCWHBZW6HWXC5TXQ, seg_pos={0 0}, upload=560833]: Long tail cancellation due to optimal threshold Jan 1 07:32:15.095362334 (640h3m14.96644537s) [noise=true , address=145.79.227.49:4338 , piece_id=BM5YJLW6ORBDFBJ5EY2YAK3G6C6NYVOIHTJZ66URITTTSQPNBL7A, seg_pos={0 0}, upload=560833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 07:32:15.095368586 (640h3m14.966451621s) [noise=true , address=145.79.227.49:4338 , piece_id=BM5YJLW6ORBDFBJ5EY2YAK3G6C6NYVOIHTJZ66URITTTSQPNBL7A, seg_pos={0 0}, upload=560833]: Long tail cancellation due to optimal threshold Jan 1 07:32:15.095497662 (640h3m14.966580707s) [noise=true , address=77.90.31.18:4808 , piece_id=JJQ2WF2AKE7MIZ23SNVIZQ5GLZ5OEYKEOMARKHZUBWCE3AF4BAYA, seg_pos={0 0}, upload=560833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:51612->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:51612->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51612->77.90.31.18:4808: use of closed network connection Jan 1 07:32:15.095518582 (640h3m14.966601617s) [noise=true , address=77.90.31.18:4808 , piece_id=JJQ2WF2AKE7MIZ23SNVIZQ5GLZ5OEYKEOMARKHZUBWCE3AF4BAYA, seg_pos={0 0}, upload=560833]: Long tail cancellation due to optimal threshold Jan 1 07:32:15.095555041 (640h3m14.966638086s) [seg_pos={0 0}, upload=560833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:32:15.095678126 (640h3m14.966761161s) [upload=560833]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:32:15.095684959 (640h3m14.966767994s) [upload=560833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:32:15.383970715 (640h3m15.25505375s) [upload=560833]: Done uploading segments... Jan 1 07:32:15.383989541 (640h3m15.255072586s) [upload=560833]: Upload finished. err: <nil> Duration: 30.003007742s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXX.XXX.XXXXXXXX.XXX.XXXXX.XX.......XX.XX.X.XXXXX.XX..XX] duration: 5.00044742s trace: https://jaeger.dev.storj.io/trace/54a2279e93f42d57 FAIL (2026-01-01 07:32:37 UTC) orbiter warning: context deadline exceeded Duration: 5.000501623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXX.XX........X...X..XXXX.....X.X...X...X.X.XX.XX..X...X.X] duration: 30.003037001s trace: https://jaeger.dev.storj.io/trace/1d8eea0b24ec742 FAIL (2026-01-01 07:32:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: PH55RVPBJ24V5M5U6GRLJXUNNWOBOKAND6MKGSSEMYHAKQ3F3THQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: RWD667HXGQHKBFLRNUBIK3GZB2ATXOZKTN5HHJH6MFXVUU4DHOFA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: T6AGMRQDVUQLJ6VOIVIO3W54WS33KVPSZWORISUOCNCOB2RPGQMA) context deadline exceeded ======LOGS====== Jan 1 07:31:41.944748485 (640h2m41.81583152s) [upload=560825]: Starting upload Jan 1 07:31:41.944807817 (640h2m41.815890862s) [upload=560825]: Done starting upload Jan 1 07:31:41.944941762 (640h2m41.816024807s) [upload=560825]: Uploading segments... Jan 1 07:31:41.945017055 (640h2m41.81610009s) [upload=560825]: Got next segment. Inline: false Jan 1 07:31:41.945049036 (640h2m41.816132071s) [upload=560825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 07:31:41.945054648 (640h2m41.816137683s) [upload=560825]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 07:31:42.192573729 (640h2m42.063656764s) [seg_pos={0 0}, upload=560825]: Begin upload segment... Jan 1 07:31:42.192642650 (640h2m42.063725695s) [seg_pos={0 0}, upload=560825]: Done begin upload segment. Jan 1 07:31:42.192702985 (640h2m42.06378603s) [noise=true , address=77.90.31.10:4518 , piece_id=T6AGMRQDVUQLJ6VOIVIO3W54WS33KVPSZWORISUOCNCOB2RPGQMA, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.192729866 (640h2m42.063812901s) [noise=true , address=145.79.227.41:4086 , piece_id=LQ6KOJAUVFBR3DTLRKBLZMHP6ONW2EC5HGMDM6UBW4QPRQZ2YIZQ, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.193329969 (640h2m42.064413014s) [noise=true , address=145.79.227.25:4619 , piece_id=NKYGNMRVNGJZA2JU6LENDVIPKIXTSJQOCSW6RZPPNEXS2CALLMTA, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.194021886 (640h2m42.065104932s) [noise=true , address=145.79.227.11:4731 , piece_id=QKS23BK2QGV657KZEG4E5KEUIUTGZNHPLXFGESA45KOS7P2AQOYA, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.194434252 (640h2m42.065517297s) [noise=true , address=145.79.227.16:4909 , piece_id=OE6DBALBQEOQU7GKW5HUGYD5QO67TH5FA7O2F5H2B77GMIB3PJEQ, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.194823703 (640h2m42.065906748s) [noise=true , address=145.79.227.15:4874 , piece_id=MB7ANHPZCBHQ7CA5LE7UOVRBGCZQDTBNVTIMQS7ALFSNHAKQHA6A, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.195199780 (640h2m42.066282815s) [noise=true , address=45.91.116.7:4012 , piece_id=VQ6JJTC7XXF2D3K6T7COSY26LENXTY6R76BVF5V6TP6GJBSE4YSA, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.196644149 (640h2m42.067727184s) [noise=true , address=45.91.116.6:4975 , piece_id=5IVOHEPIUJUYY6K7CSWQ5V6A7CKUPIFYOY3RVU2ENPQMVQ23CNPA, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.196678034 (640h2m42.067761069s) [noise=true , address=145.79.227.24:4583 , piece_id=GM3AAFLNZX4WSGSN432HADL7UT57PCQ425LUOTPW3A5CJUI6KYDQ, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.198093419 (640h2m42.069176454s) [noise=true , address=145.79.227.10:4403 , piece_id=BJ2DPWGYN3Y7DCF4PY35RBGNWNDSQIK6SAJFOKK4CJC2ZATV5AGQ, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.198617477 (640h2m42.069700522s) [noise=true , address=145.79.227.4:4187 , piece_id=PTS6OATU3KRK3RVITKQUN4BEVM33HGNJL2H23LPHRLFR5O2SJW2Q, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.199211839 (640h2m42.070294874s) [noise=true , address=45.91.116.11:4118 , piece_id=U2H3X563RIFXSJRA3NO7NIVCLPO3GW5AZEV4ISQT55NG3UAFLMKQ, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.200260145 (640h2m42.07134318s) [noise=true , address=77.90.31.12:4589 , piece_id=PH55RVPBJ24V5M5U6GRLJXUNNWOBOKAND6MKGSSEMYHAKQ3F3THQ, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.200618898 (640h2m42.071701933s) [noise=true , address=145.79.227.39:4013 , piece_id=WAFIFARSJCMCJCYZHSA5N7NLFSVELYXSX65T37HEPP4RETMY2JUQ, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.201452094 (640h2m42.072535139s) [noise=true , address=45.91.116.10:4084 , piece_id=RWD667HXGQHKBFLRNUBIK3GZB2ATXOZKTN5HHJH6MFXVUU4DHOFA, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:42.202721873 (640h2m42.073804908s) [noise=true , address=145.79.227.50:4374 , piece_id=6TO24Y5BZKBDRMBKBHXDKXDIURBSQGNBQNMTZR6FI5PBAEEVFIUA, seg_pos={0 0}, upload=560825]: Uploading piece... Jan 1 07:31:43.882358585 (640h2m43.75344162s) [upload=560825]: Next returned nil segment Jan 1 07:31:43.882384174 (640h2m43.753467219s) [upload=560825]: Waiting for error group managing segments... Jan 1 07:31:44.191822277 (640h2m44.062905322s) [noise=true , address=145.79.227.16:4909 , piece_id=OE6DBALBQEOQU7GKW5HUGYD5QO67TH5FA7O2F5H2B77GMIB3PJEQ, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.198349637 (640h2m44.069432672s) [noise=true , address=145.79.227.50:4374 , piece_id=6TO24Y5BZKBDRMBKBHXDKXDIURBSQGNBQNMTZR6FI5PBAEEVFIUA, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.201340892 (640h2m44.072423937s) [noise=true , address=145.79.227.41:4086 , piece_id=LQ6KOJAUVFBR3DTLRKBLZMHP6ONW2EC5HGMDM6UBW4QPRQZ2YIZQ, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.201708041 (640h2m44.072791076s) [noise=true , address=145.79.227.24:4583 , piece_id=GM3AAFLNZX4WSGSN432HADL7UT57PCQ425LUOTPW3A5CJUI6KYDQ, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.213844861 (640h2m44.084927896s) [noise=true , address=145.79.227.4:4187 , piece_id=PTS6OATU3KRK3RVITKQUN4BEVM33HGNJL2H23LPHRLFR5O2SJW2Q, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.218418089 (640h2m44.089501124s) [noise=true , address=145.79.227.39:4013 , piece_id=WAFIFARSJCMCJCYZHSA5N7NLFSVELYXSX65T37HEPP4RETMY2JUQ, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.221093884 (640h2m44.092176929s) [noise=true , address=145.79.227.25:4619 , piece_id=NKYGNMRVNGJZA2JU6LENDVIPKIXTSJQOCSW6RZPPNEXS2CALLMTA, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.277112341 (640h2m44.148195376s) [noise=true , address=145.79.227.10:4403 , piece_id=BJ2DPWGYN3Y7DCF4PY35RBGNWNDSQIK6SAJFOKK4CJC2ZATV5AGQ, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.277573610 (640h2m44.148656645s) [noise=true , address=145.79.227.11:4731 , piece_id=QKS23BK2QGV657KZEG4E5KEUIUTGZNHPLXFGESA45KOS7P2AQOYA, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.277961759 (640h2m44.149044794s) [noise=true , address=145.79.227.15:4874 , piece_id=MB7ANHPZCBHQ7CA5LE7UOVRBGCZQDTBNVTIMQS7ALFSNHAKQHA6A, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.384802769 (640h2m44.255885804s) [noise=true , address=77.90.31.12:4589 , piece_id=PH55RVPBJ24V5M5U6GRLJXUNNWOBOKAND6MKGSSEMYHAKQ3F3THQ, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:44.423104160 (640h2m44.294187195s) [noise=true , address=77.90.31.10:4518 , piece_id=T6AGMRQDVUQLJ6VOIVIO3W54WS33KVPSZWORISUOCNCOB2RPGQMA, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:45.172146782 (640h2m45.043229817s) [noise=true , address=45.91.116.10:4084 , piece_id=RWD667HXGQHKBFLRNUBIK3GZB2ATXOZKTN5HHJH6MFXVUU4DHOFA, seg_pos={0 0}, upload=560825]: Done uploading piece. err: <nil> Jan 1 07:31:45.172205574 (640h2m45.043288619s) [seg_pos={0 0}, upload=560825]: Segment reached optimal threshold of 13 pieces. Jan 1 07:31:45.172453716 (640h2m45.043536761s) [noise=true , address=45.91.116.6:4975 , piece_id=5IVOHEPIUJUYY6K7CSWQ5V6A7CKUPIFYOY3RVU2ENPQMVQ23CNPA, seg_pos={0 0}, upload=560825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:45.172466291 (640h2m45.043549336s) [noise=true , address=45.91.116.6:4975 , piece_id=5IVOHEPIUJUYY6K7CSWQ5V6A7CKUPIFYOY3RVU2ENPQMVQ23CNPA, seg_pos={0 0}, upload=560825]: Long tail cancellation due to optimal threshold Jan 1 07:31:45.173001279 (640h2m45.044084324s) [noise=true , address=45.91.116.7:4012 , piece_id=VQ6JJTC7XXF2D3K6T7COSY26LENXTY6R76BVF5V6TP6GJBSE4YSA, seg_pos={0 0}, upload=560825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:41882->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:41882->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41882->45.91.116.7:4012: use of closed network connection Jan 1 07:31:45.173011378 (640h2m45.044094423s) [noise=true , address=45.91.116.7:4012 , piece_id=VQ6JJTC7XXF2D3K6T7COSY26LENXTY6R76BVF5V6TP6GJBSE4YSA, seg_pos={0 0}, upload=560825]: Long tail cancellation due to optimal threshold Jan 1 07:31:45.173143610 (640h2m45.044226645s) [noise=true , address=45.91.116.11:4118 , piece_id=U2H3X563RIFXSJRA3NO7NIVCLPO3GW5AZEV4ISQT55NG3UAFLMKQ, seg_pos={0 0}, upload=560825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:45.173151826 (640h2m45.044234861s) [noise=true , address=45.91.116.11:4118 , piece_id=U2H3X563RIFXSJRA3NO7NIVCLPO3GW5AZEV4ISQT55NG3UAFLMKQ, seg_pos={0 0}, upload=560825]: Long tail cancellation due to optimal threshold Jan 1 07:31:45.173180160 (640h2m45.044263205s) [seg_pos={0 0}, upload=560825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 07:31:45.173324895 (640h2m45.04440793s) [upload=560825]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:31:45.173331788 (640h2m45.044414833s) [upload=560825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:31:45.469945782 (640h2m45.341028837s) [upload=560825]: Done uploading segments... Jan 1 07:31:45.469970469 (640h2m45.341053504s) [upload=560825]: Upload finished. err: <nil> Duration: 30.003060465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX............X...........X.....................X.....X] duration: 1m0.003697349s trace: https://jaeger.dev.storj.io/trace/70a1e85d41a2cf70 FAIL (2026-01-01 07:32:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== MFIPHRKDSXY47IDNA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.864997229 (640h2m7.736080274s) [noise=true , address=145.79.227.17:4999 , piece_id=JZUMG5W7HHUB5MKPHGV5QOA4HI63MHQGGW43RBHXQGARNHUM7S4Q, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.865516548 (640h2m7.736599583s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866053551 (640h2m7.737136596s) [noise=true , address=145.79.227.52:4454 , piece_id=ZNRPDZXJW6NWPEQRZVFFECBQYJSTBTIAQZ43JRHB3KHT5RBWWTCA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866066876 (640h2m7.737149911s) [noise=true , address=145.79.227.40:4071 , piece_id=JSBG2ASARQZR7OAZUXNMWSHIHTEXFCB23DVHRWZFVNFBHPJEV4RQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866484080 (640h2m7.737567125s) [noise=true , address=77.90.31.14:4669 , piece_id=P5QENHQ7JQKVJYQLKQ4O3UA4S3MZKUTE6KRQF5KBC7VA226LX4TQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866554494 (640h2m7.73763754s) [noise=true , address=77.90.31.14:4696 , piece_id=3G3FLTWQY2LBJML7SLPVTUSBSZGACHPTCWFDIPGCOSIEOIPPSPTQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866878461 (640h2m7.737961506s) [noise=true , address=145.79.227.10:4412 , piece_id=G2HHTDKWAC723SXOTU77RSBXCJ5U2X7DV6MSEHCK7RXYQZKPBZ5A, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.866927014 (640h2m7.738010049s) [noise=true , address=145.79.227.4:4192 , piece_id=5PMVDGHG54LQH7JQIBPZLXRZPCYMW2SFBNHKGVPCWLH4SKIDZWRA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.867276009 (640h2m7.738359054s) [noise=true , address=145.79.227.44:4211 , piece_id=BXZ3AUSEGUWVHGGBRL52UCB5253AEH6BCUBC7LLCSHZDZQEAPZLQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.867705596 (640h2m7.738788641s) [noise=true , address=145.79.227.49:4365 , piece_id=ZWEY36GNU5MRECJAYWI6VMTMPEAC3H6WEUATUUPSEYM3LKKY6FBA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868116399 (640h2m7.739199444s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868541698 (640h2m7.739624733s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.868737772 (640h2m7.739820807s) [noise=true , address=45.91.116.6:4006 , piece_id=JT64JIMBSVQJKZLVYTJB6UG36IJYEJ5ZWQPVEONVGOKHTM27IJIQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.869182529 (640h2m7.740265564s) [noise=true , address=145.79.227.29:4949 , piece_id=HV4NWKEXPQZ2JMZ2DKWAC5F5E46GOMRJRV5JYXGY4H4F24GIGMOA, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.869774086 (640h2m7.740857131s) [noise=true , address=64.49.13.26:4955 , piece_id=II6N2UZV5NRFLGNFVPMEW3Q3ZBSIMEKWXLNGQ34LCLUBGTTCTKQQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.870263738 (640h2m7.741346783s) [noise=true , address=77.90.31.21:4887 , piece_id=54SKDJSIGOY2S6CAGHZSHRFQP5POMZLTCO6H2CAHPFLQOFN7ZY2A, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:07.870718674 (640h2m7.741801719s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Uploading piece... Jan 1 07:31:09.286406211 (640h2m9.157489246s) [upload=560816]: Next returned nil segment Jan 1 07:31:09.286424576 (640h2m9.157507611s) [upload=560816]: Waiting for error group managing segments... Jan 1 07:31:09.646470543 (640h2m9.517553588s) [noise=true , address=64.49.13.26:4955 , piece_id=II6N2UZV5NRFLGNFVPMEW3Q3ZBSIMEKWXLNGQ34LCLUBGTTCTKQQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.729572180 (640h2m9.600655215s) [noise=true , address=145.79.227.29:4949 , piece_id=HV4NWKEXPQZ2JMZ2DKWAC5F5E46GOMRJRV5JYXGY4H4F24GIGMOA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.756915146 (640h2m9.627998191s) [noise=true , address=145.79.227.4:4192 , piece_id=5PMVDGHG54LQH7JQIBPZLXRZPCYMW2SFBNHKGVPCWLH4SKIDZWRA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.773518300 (640h2m9.644601335s) [noise=true , address=145.79.227.10:4412 , piece_id=G2HHTDKWAC723SXOTU77RSBXCJ5U2X7DV6MSEHCK7RXYQZKPBZ5A, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.813080993 (640h2m9.684164038s) [noise=true , address=145.79.227.17:4999 , piece_id=JZUMG5W7HHUB5MKPHGV5QOA4HI63MHQGGW43RBHXQGARNHUM7S4Q, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.898457802 (640h2m9.769540837s) [noise=true , address=77.90.31.14:4696 , piece_id=3G3FLTWQY2LBJML7SLPVTUSBSZGACHPTCWFDIPGCOSIEOIPPSPTQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:09.989265563 (640h2m9.860348608s) [noise=true , address=145.79.227.40:4071 , piece_id=JSBG2ASARQZR7OAZUXNMWSHIHTEXFCB23DVHRWZFVNFBHPJEV4RQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.004850749 (640h2m9.875933795s) [noise=true , address=77.90.31.14:4669 , piece_id=P5QENHQ7JQKVJYQLKQ4O3UA4S3MZKUTE6KRQF5KBC7VA226LX4TQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.022329011 (640h2m9.893412056s) [noise=true , address=145.79.227.52:4454 , piece_id=ZNRPDZXJW6NWPEQRZVFFECBQYJSTBTIAQZ43JRHB3KHT5RBWWTCA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.077129869 (640h2m9.948212914s) [noise=true , address=77.90.31.21:4887 , piece_id=54SKDJSIGOY2S6CAGHZSHRFQP5POMZLTCO6H2CAHPFLQOFN7ZY2A, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.116925085 (640h2m9.98800812s) [noise=true , address=45.91.116.6:4006 , piece_id=JT64JIMBSVQJKZLVYTJB6UG36IJYEJ5ZWQPVEONVGOKHTM27IJIQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.620867179 (640h2m10.491950214s) [noise=true , address=145.79.227.35:4494 , piece_id=VVD37LPMV523QCTFI2FR3YW3MGGY23VWEB6MFIPHRKDSXY47IDNA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.624754562 (640h2m10.495837607s) [noise=true , address=145.79.227.49:4365 , piece_id=ZWEY36GNU5MRECJAYWI6VMTMPEAC3H6WEUATUUPSEYM3LKKY6FBA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.625559685 (640h2m10.49664273s) [noise=true , address=145.79.227.44:4211 , piece_id=BXZ3AUSEGUWVHGGBRL52UCB5253AEH6BCUBC7LLCSHZDZQEAPZLQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: <nil> Jan 1 07:31:10.625581687 (640h2m10.496664732s) [seg_pos={0 0}, upload=560816]: Segment reached optimal threshold of 14 pieces. Jan 1 07:31:10.625942523 (640h2m10.497025599s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.625951421 (640h2m10.497034456s) [noise=true , address=193.149.164.143:4695 , piece_id=B2CLRYIH7VAMVWZFMCJMBZMIIIAHEWDBDH774SOYPCEGFYXAUHRQ, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.625964065 (640h2m10.4970471s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.625973132 (640h2m10.497056177s) [noise=true , address=145.79.227.30:4783 , piece_id=F7JYZ5ZVFKZFIPPREZ4KXI4WETYZITCNA5SNKKEFAX5ZUPSJYIDA, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626108109 (640h2m10.497191144s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.626115994 (640h2m10.497199029s) [noise=true , address=145.79.227.33:4877 , piece_id=GNB2LDALR2SKXLOBVWE6YLDEAOCVGGT4USZE5W5EUVAEUQUZEAWQ, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626251302 (640h2m10.497334347s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PRN34y7cCWokJkxgu5Zk4eBjDPW5iddNefMKXvUkJUTAqkKve): protocol: expected piece hash; context canceled; EOF Jan 1 07:31:10.626258255 (640h2m10.4973413s) [noise=true , address=45.91.116.4:4958 , piece_id=GQENWJTVBA6RJOZB65BCYJVTYLQSQHUK7LEB2ORT7FDZESYIX5IA, seg_pos={0 0}, upload=560816]: Long tail cancellation due to optimal threshold Jan 1 07:31:10.626318860 (640h2m10.497401905s) [seg_pos={0 0}, upload=560816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 07:31:10.626397239 (640h2m10.497480284s) [upload=560816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 07:31:10.626402520 (640h2m10.497485565s) [upload=560816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 07:31:10.922618708 (640h2m10.793701743s) [upload=560816]: Done uploading segments... Jan 1 07:31:10.922646190 (640h2m10.793729225s) [upload=560816]: Upload finished. err: <nil> Duration: 1m0.003710274s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)