Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X..........] duration: 6.151023262s FAIL (2025-09-22 02:46:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.151060192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....XXXXX...XX..............................................] duration: 5.905129838s FAIL (2025-09-22 02:46:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.905159595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X.........] duration: 6.097274956s FAIL (2025-09-22 02:46:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.097331935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....XXXXX...XX.............................................] duration: 5.965842033s FAIL (2025-09-22 02:45:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.965873133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..............XX........X.........X.....XXX.X.X.XXXXX..X.X..] duration: 6.053296745s FAIL (2025-09-22 02:45:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.053361919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......XXXXX...XX............................................] duration: 6.079696544s FAIL (2025-09-22 02:45:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.079808708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......XXXXX...XX...........................................] duration: 6.021373595s FAIL (2025-09-22 02:45:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.02141775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [......XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X........] duration: 6.177268769s FAIL (2025-09-22 02:45:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.177313655s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...............XX........X.........X.....XXX.X.X.XXXXX..X.X.] duration: 6.072505992s FAIL (2025-09-22 02:45:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.072538473s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........XXXXX...XX..........................................] duration: 5.950673701s FAIL (2025-09-22 02:44:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.950706122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........XXXXX...XX.........................................] duration: 6.038667096s FAIL (2025-09-22 02:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.038695682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [................XX........X.........X.....XXX.X.X.XXXXX..X.X] duration: 30.001103377s FAIL (2025-09-22 02:44:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001134515s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.......XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X.......] duration: 5.992105385s FAIL (2025-09-22 02:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.992137507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........XXXXX...XX........................................] duration: 5.994287208s FAIL (2025-09-22 02:44:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.994322063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................XX........X.........X.....XXX.X.X.XXXXX..X.] duration: 5.954652386s FAIL (2025-09-22 02:43:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.954684757s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........XXXXX...XX.......................................] duration: 6.020981058s FAIL (2025-09-22 02:43:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.02109719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [........XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X......] duration: 6.234930587s FAIL (2025-09-22 02:43:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.23496339s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................XX........X.........X.....XXX.X.X.XXXXX..X] duration: 30.00120063s FAIL (2025-09-22 02:42:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001233002s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............XXXXX...XX......................................] duration: 6.042137343s FAIL (2025-09-22 02:42:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.042171309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.........XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X.....] duration: 6.022577218s FAIL (2025-09-22 02:42:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.022622043s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............XXXXX...XX.....................................] duration: 6.386190623s FAIL (2025-09-22 02:41:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.386227894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............XXXXX...XX....................................] duration: 6.016581973s FAIL (2025-09-22 02:41:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.016606199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................XX........X.........X.....XXX.X.X.XXXXX..] duration: 6.167384079s FAIL (2025-09-22 02:41:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.1675269s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..........XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X....] duration: 6.333110452s FAIL (2025-09-22 02:41:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.333144457s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................XX........X.........X.....XXX.X.X.XXXXX.] duration: 9.554681907s FAIL (2025-09-22 02:41:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.55470987s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....................XX........X.........X.....XXX.X.X.XXXXX] duration: 30.001085232s FAIL (2025-09-22 02:40:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001114457s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...........XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X...] duration: 6.107728993s FAIL (2025-09-22 02:40:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.107767435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............XXXXX...XX...................................] duration: 6.070742082s FAIL (2025-09-22 02:40:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.070776979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......................XX........X.........X.....XXX.X.X.XXXX] duration: 30.001004618s FAIL (2025-09-22 02:40:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0010327s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [............XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X..] duration: 6.410214197s FAIL (2025-09-22 02:40:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.410250467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [................XXXXX...XX..................................] duration: 5.734170879s FAIL (2025-09-22 02:39:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.734201126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......................XX........X.........X.....XXX.X.X.XXX] duration: 30.000745137s FAIL (2025-09-22 02:39:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000846039s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................XXXXX...XX.................................] duration: 6.236774357s FAIL (2025-09-22 02:39:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.236809706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.............XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X.] duration: 6.374228975s FAIL (2025-09-22 02:39:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.374250726s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........................XX........X.........X.....XXX.X.X.XX] duration: 30.001601178s FAIL (2025-09-22 02:39:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001631125s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..............XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...X] duration: 30.243319847s FAIL (2025-09-22 02:38:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:38:29.800724052 (373h10m5.840297182s) [upload=327992]: Starting upload Sep 22 02:38:29.800820897 (373h10m5.840394027s) [upload=327992]: Done starting upload Sep 22 02:38:29.800982605 (373h10m5.840555735s) [upload=327992]: Uploading segments... Sep 22 02:38:29.801109617 (373h10m5.840682747s) [upload=327992]: Got next segment. Inline: false Sep 22 02:38:29.801142400 (373h10m5.84071553s) [upload=327992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:38:29.801147609 (373h10m5.840720739s) [upload=327992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:38:30.053487181 (373h10m6.093060311s) [seg_pos={0 0}, upload=327992]: Begin upload segment... Sep 22 02:38:30.053547055 (373h10m6.093120175s) [seg_pos={0 0}, upload=327992]: Done begin upload segment. Sep 22 02:38:30.053581140 (373h10m6.09315426s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.053630574 (373h10m6.093203714s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054175792 (373h10m6.093748922s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054186602 (373h10m6.093759732s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054538352 (373h10m6.094111472s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.054883650 (373h10m6.09445678s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055064594 (373h10m6.094637724s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055282569 (373h10m6.094855689s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055575447 (373h10m6.095148567s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.055965219 (373h10m6.095538349s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056062354 (373h10m6.095635484s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056414064 (373h10m6.095987184s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056602743 (373h10m6.096175863s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.056963520 (373h10m6.09653665s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057315420 (373h10m6.09688856s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:30.057642192 (373h10m6.097215322s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Uploading piece... Sep 22 02:38:31.814197640 (373h10m7.85377077s) [upload=327992]: Next returned nil segment Sep 22 02:38:31.814225273 (373h10m7.853798393s) [upload=327992]: Waiting for error group managing segments... Sep 22 02:38:32.093033916 (373h10m8.132607046s) [noise=true , address=45.91.116.13:4155 , piece_id=IBYSIPONHVUYOQPHYDAITFSOUBWABGAZUJQYV2X6VZVBHSQ6ETMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.136060256 (373h10m8.175633376s) [noise=true , address=77.90.31.17:4771 , piece_id=UPFW6WIL7JEF7ES56HOU2UUSQEZEBGVLV2KPX5EAQ2375WHWH3JQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.148836585 (373h10m8.188409705s) [noise=true , address=45.91.116.9:4261 , piece_id=WG7AXI7KZVENNTF2M5BH7K7YSEJJHSEQVZ7Z4ZQLLRKWVY4MXUGQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.152011741 (373h10m8.191584861s) [noise=true , address=77.90.31.13:4626 , piece_id=3MRECYKB6VMU3YM4GJWMYPO2QH53EYUBUBXFOEADVRLLJ6VIFKBQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.160342644 (373h10m8.199915774s) [noise=true , address=45.91.116.7:4011 , piece_id=LOSX7IN6ANM6D34PHJRWTTZXFHHWCWS34RXZHRPXS3OHP6OIAEEA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.171925390 (373h10m8.21149852s) [noise=true , address=45.91.116.8:4047 , piece_id=DAFTQJXR3BJ7AQM7X6DV4H5WYCMHUXOMSRSFPMLJBT6EBDWEENMQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.194514263 (373h10m8.234087383s) [noise=true , address=77.90.31.18:4806 , piece_id=AER5QXCSR5LLLXMSIJZGQ63X4WTBSNNYHUAJODPSQ75SYSGEVSYQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.202461106 (373h10m8.242034226s) [noise=true , address=45.91.116.10:4083 , piece_id=V2FUTFXWBXMSMRYRBHTUK3JRCXEQHJFDOAIX24XUJ35U6DNL6NAQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.234771400 (373h10m8.27434452s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZLRNPDOAHD4EOTFV34FXEKBJ7OXVP4SL753TZBMCWKADMLB46FA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.257978711 (373h10m8.297551841s) [noise=true , address=77.90.31.14:4663 , piece_id=X5Z7CUB2TYPK6EISHTH4D5D4FM2EWPTQ7L3PPCYKQ2C2WBXELFMA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.263925134 (373h10m8.303498254s) [noise=true , address=77.90.31.21:4879 , piece_id=7IU4BVIMOUFI2S2EYJ3AU2B4IJZPMF74JJQTIL2H743COERL7GIA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:32.332829320 (373h10m8.37240245s) [noise=true , address=45.91.116.5:4299 , piece_id=UBLL7OWH33ETQYCWQ4J35E5TOO5QJ2NFPWH2Z4TZYM3CEMKH44QQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: <nil> Sep 22 02:38:59.090751403 (373h10m35.130324523s) [noise=true , address=145.79.227.14:4838 , piece_id=R3K4BBPRINWOFQMMNO3W6EXDNMAK6RU57UNKDVRF52DRJCF6RSSQ, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090780318 (373h10m35.130353438s) [noise=true , address=145.79.227.7:4293 , piece_id=L2CWKVBOEIDYT72GLKOTB25E5NBXZP4LBSJK6STNGBV6BZFN6OZA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090811568 (373h10m35.130384698s) [noise=true , address=145.79.227.19:4078 , piece_id=4JE6ZYICHOBJM57JBGMGYAYIWLG7TIW4T7F7XHYBTVVYWCPIHMLA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090851323 (373h10m35.130424444s) [noise=true , address=145.79.227.15:4875 , piece_id=3RTIOAMVAG6TAQICJEFA3W47BPIK4CGY7C47UEIVPGLF5UUZWIVA, seg_pos={0 0}, upload=327992]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:38:59.090888023 (373h10m35.130461153s) [seg_pos={0 0}, upload=327992]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:38:59.332384338 (373h10m35.371957458s) [upload=327992]: Done uploading segments... Sep 22 02:38:59.332413593 (373h10m35.371986713s) [upload=327992]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.243346076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................XXXXX...XX................................] duration: 6.057693884s FAIL (2025-09-22 02:38:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.057742687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...............XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX...] duration: 6.224466168s FAIL (2025-09-22 02:38:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:36:06.836593868 (373h7m42.876166998s) [upload=327956]: Starting upload Sep 22 02:36:06.836660575 (373h7m42.876233695s) [upload=327956]: Done starting upload Sep 22 02:36:06.836888098 (373h7m42.876461228s) [upload=327956]: Uploading segments... Sep 22 02:36:06.836936070 (373h7m42.8765092s) [upload=327956]: Got next segment. Inline: false Sep 22 02:36:06.836978521 (373h7m42.876551651s) [upload=327956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:36:06.836985614 (373h7m42.876558744s) [upload=327956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:36:07.096866435 (373h7m43.136439565s) [seg_pos={0 0}, upload=327956]: Begin upload segment... Sep 22 02:36:07.096989920 (373h7m43.13656304s) [seg_pos={0 0}, upload=327956]: Done begin upload segment. Sep 22 02:36:07.097016541 (373h7m43.136589661s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097030749 (373h7m43.136603879s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097640800 (373h7m43.13721394s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097685104 (373h7m43.137258234s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098145682 (373h7m43.137718812s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098235522 (373h7m43.137808652s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098496269 (373h7m43.138069399s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098748730 (373h7m43.13832186s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098973046 (373h7m43.138546176s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.099479832 (373h7m43.139052962s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100028076 (373h7m43.139601196s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100442144 (373h7m43.140015264s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100675097 (373h7m43.140248227s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101073947 (373h7m43.140647067s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101452478 (373h7m43.141025608s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101874963 (373h7m43.141448093s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:08.830620424 (373h7m44.870193554s) [upload=327956]: Next returned nil segment Sep 22 02:36:08.830643749 (373h7m44.870216879s) [upload=327956]: Waiting for error group managing segments... Sep 22 02:36:09.159306203 (373h7m45.198879334s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.167498683 (373h7m45.207071803s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.182258188 (373h7m45.221831308s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.214465988 (373h7m45.254039118s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.222380368 (373h7m45.261953498s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.224530151 (373h7m45.264103282s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.234179205 (373h7m45.273752335s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.261465163 (373h7m45.301038283s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.279267742 (373h7m45.318840872s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.287519997 (373h7m45.327093127s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.301046413 (373h7m45.340619543s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.342349692 (373h7m45.381922812s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:36.120746520 (373h8m12.16031964s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:36.120818356 (373h8m12.160391486s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120876898 (373h8m12.160450018s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120944036 (373h8m12.160517156s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120983501 (373h8m12.160556621s) [seg_pos={0 0}, upload=327956]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:36:36.368177143 (373h8m12.407750263s) [upload=327956]: Done uploading segments... Sep 22 02:36:36.368209475 (373h8m12.407782595s) [upload=327956]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.224499432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................XXXXX...XX...............................] duration: 6.078711232s FAIL (2025-09-22 02:38:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.078732223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................XX........X.........X.....XXX.X.X.X] duration: 30.001837445s FAIL (2025-09-22 02:38:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001857935s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [................XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX..] duration: 6.038036161s FAIL (2025-09-22 02:37:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:36:06.836593868 (373h7m42.876166998s) [upload=327956]: Starting upload Sep 22 02:36:06.836660575 (373h7m42.876233695s) [upload=327956]: Done starting upload Sep 22 02:36:06.836888098 (373h7m42.876461228s) [upload=327956]: Uploading segments... Sep 22 02:36:06.836936070 (373h7m42.8765092s) [upload=327956]: Got next segment. Inline: false Sep 22 02:36:06.836978521 (373h7m42.876551651s) [upload=327956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:36:06.836985614 (373h7m42.876558744s) [upload=327956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:36:07.096866435 (373h7m43.136439565s) [seg_pos={0 0}, upload=327956]: Begin upload segment... Sep 22 02:36:07.096989920 (373h7m43.13656304s) [seg_pos={0 0}, upload=327956]: Done begin upload segment. Sep 22 02:36:07.097016541 (373h7m43.136589661s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097030749 (373h7m43.136603879s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097640800 (373h7m43.13721394s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097685104 (373h7m43.137258234s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098145682 (373h7m43.137718812s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098235522 (373h7m43.137808652s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098496269 (373h7m43.138069399s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098748730 (373h7m43.13832186s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098973046 (373h7m43.138546176s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.099479832 (373h7m43.139052962s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100028076 (373h7m43.139601196s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100442144 (373h7m43.140015264s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100675097 (373h7m43.140248227s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101073947 (373h7m43.140647067s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101452478 (373h7m43.141025608s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101874963 (373h7m43.141448093s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:08.830620424 (373h7m44.870193554s) [upload=327956]: Next returned nil segment Sep 22 02:36:08.830643749 (373h7m44.870216879s) [upload=327956]: Waiting for error group managing segments... Sep 22 02:36:09.159306203 (373h7m45.198879334s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.167498683 (373h7m45.207071803s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.182258188 (373h7m45.221831308s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.214465988 (373h7m45.254039118s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.222380368 (373h7m45.261953498s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.224530151 (373h7m45.264103282s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.234179205 (373h7m45.273752335s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.261465163 (373h7m45.301038283s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.279267742 (373h7m45.318840872s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.287519997 (373h7m45.327093127s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.301046413 (373h7m45.340619543s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.342349692 (373h7m45.381922812s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:36.120746520 (373h8m12.16031964s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:36.120818356 (373h8m12.160391486s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120876898 (373h8m12.160450018s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120944036 (373h8m12.160517156s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120983501 (373h8m12.160556621s) [seg_pos={0 0}, upload=327956]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:36:36.368177143 (373h8m12.407750263s) [upload=327956]: Done uploading segments... Sep 22 02:36:36.368209475 (373h8m12.407782595s) [upload=327956]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.038069815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................XXXXX...XX..............................] duration: 6.003077423s FAIL (2025-09-22 02:37:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.003115816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.................XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX.] duration: 5.914783757s FAIL (2025-09-22 02:36:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:36:06.836593868 (373h7m42.876166998s) [upload=327956]: Starting upload Sep 22 02:36:06.836660575 (373h7m42.876233695s) [upload=327956]: Done starting upload Sep 22 02:36:06.836888098 (373h7m42.876461228s) [upload=327956]: Uploading segments... Sep 22 02:36:06.836936070 (373h7m42.8765092s) [upload=327956]: Got next segment. Inline: false Sep 22 02:36:06.836978521 (373h7m42.876551651s) [upload=327956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:36:06.836985614 (373h7m42.876558744s) [upload=327956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:36:07.096866435 (373h7m43.136439565s) [seg_pos={0 0}, upload=327956]: Begin upload segment... Sep 22 02:36:07.096989920 (373h7m43.13656304s) [seg_pos={0 0}, upload=327956]: Done begin upload segment. Sep 22 02:36:07.097016541 (373h7m43.136589661s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097030749 (373h7m43.136603879s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097640800 (373h7m43.13721394s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097685104 (373h7m43.137258234s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098145682 (373h7m43.137718812s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098235522 (373h7m43.137808652s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098496269 (373h7m43.138069399s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098748730 (373h7m43.13832186s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098973046 (373h7m43.138546176s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.099479832 (373h7m43.139052962s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100028076 (373h7m43.139601196s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100442144 (373h7m43.140015264s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100675097 (373h7m43.140248227s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101073947 (373h7m43.140647067s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101452478 (373h7m43.141025608s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101874963 (373h7m43.141448093s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:08.830620424 (373h7m44.870193554s) [upload=327956]: Next returned nil segment Sep 22 02:36:08.830643749 (373h7m44.870216879s) [upload=327956]: Waiting for error group managing segments... Sep 22 02:36:09.159306203 (373h7m45.198879334s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.167498683 (373h7m45.207071803s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.182258188 (373h7m45.221831308s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.214465988 (373h7m45.254039118s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.222380368 (373h7m45.261953498s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.224530151 (373h7m45.264103282s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.234179205 (373h7m45.273752335s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.261465163 (373h7m45.301038283s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.279267742 (373h7m45.318840872s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.287519997 (373h7m45.327093127s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.301046413 (373h7m45.340619543s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.342349692 (373h7m45.381922812s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:36.120746520 (373h8m12.16031964s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:36.120818356 (373h8m12.160391486s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120876898 (373h8m12.160450018s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120944036 (373h8m12.160517156s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120983501 (373h8m12.160556621s) [seg_pos={0 0}, upload=327956]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:36:36.368177143 (373h8m12.407750263s) [upload=327956]: Done uploading segments... Sep 22 02:36:36.368209475 (373h8m12.407782595s) [upload=327956]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.914805258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..................XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XXX] duration: 30.249353503s FAIL (2025-09-22 02:36:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:36:06.836593868 (373h7m42.876166998s) [upload=327956]: Starting upload Sep 22 02:36:06.836660575 (373h7m42.876233695s) [upload=327956]: Done starting upload Sep 22 02:36:06.836888098 (373h7m42.876461228s) [upload=327956]: Uploading segments... Sep 22 02:36:06.836936070 (373h7m42.8765092s) [upload=327956]: Got next segment. Inline: false Sep 22 02:36:06.836978521 (373h7m42.876551651s) [upload=327956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:36:06.836985614 (373h7m42.876558744s) [upload=327956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:36:07.096866435 (373h7m43.136439565s) [seg_pos={0 0}, upload=327956]: Begin upload segment... Sep 22 02:36:07.096989920 (373h7m43.13656304s) [seg_pos={0 0}, upload=327956]: Done begin upload segment. Sep 22 02:36:07.097016541 (373h7m43.136589661s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097030749 (373h7m43.136603879s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097640800 (373h7m43.13721394s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.097685104 (373h7m43.137258234s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098145682 (373h7m43.137718812s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098235522 (373h7m43.137808652s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098496269 (373h7m43.138069399s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098748730 (373h7m43.13832186s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.098973046 (373h7m43.138546176s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.099479832 (373h7m43.139052962s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100028076 (373h7m43.139601196s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100442144 (373h7m43.140015264s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.100675097 (373h7m43.140248227s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101073947 (373h7m43.140647067s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101452478 (373h7m43.141025608s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:07.101874963 (373h7m43.141448093s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Uploading piece... Sep 22 02:36:08.830620424 (373h7m44.870193554s) [upload=327956]: Next returned nil segment Sep 22 02:36:08.830643749 (373h7m44.870216879s) [upload=327956]: Waiting for error group managing segments... Sep 22 02:36:09.159306203 (373h7m45.198879334s) [noise=true , address=77.90.31.21:4879 , piece_id=SQUVTQ24BMSOBXXBIBNQSEQA4KG3MPWJ6YQ6TMADU6SMQ6KZXXWQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.167498683 (373h7m45.207071803s) [noise=true , address=77.90.31.15:4697 , piece_id=4OCYNWLXAVDU32L6B6V5UOJWQM3ZV4D4E5P5FIE7I3OJDGQ4MCVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.182258188 (373h7m45.221831308s) [noise=true , address=77.90.31.12:4589 , piece_id=LU7SHR3LS42ZMW6KC53HSBH5W3ZIC5PQW7GSQ6JFV5LE5YYASOKQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.214465988 (373h7m45.254039118s) [noise=true , address=45.91.116.8:4045 , piece_id=AMI4BOBDIMETU5R6Y2FL5LSLIYH45JSQMC4WEW7XLXWZ56EJGQBA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.222380368 (373h7m45.261953498s) [noise=true , address=45.91.116.11:4117 , piece_id=E6NQ5IGRUBUSRIFQFCVI6P5JYKRN6WQQWPKGLY3ELA5IWDOZVBZQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.224530151 (373h7m45.264103282s) [noise=true , address=45.91.116.14:4191 , piece_id=CFN3ZVCDVZ2PI2NBKWJLYVQ6NEP4QFO6BGSFGLKAIYSQPK2JBEAA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.234179205 (373h7m45.273752335s) [noise=true , address=45.91.116.10:4081 , piece_id=RX554VX4WUIVVFYZK7Q62BPZBYUA5I77SSA3J27SKI7BR7TN3IGQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.261465163 (373h7m45.301038283s) [noise=true , address=45.91.116.12:4225 , piece_id=RP5A26VTATQFJYJOSEDM2EXDAWCG6YOD6BB53EFB5MTQMK5L7IIQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.279267742 (373h7m45.318840872s) [noise=true , address=77.90.31.13:4626 , piece_id=VZU6VK6BEYZDXMAGGRRDTF76GO22A63JIXL6LPP5OZOUP2H2UWTA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.287519997 (373h7m45.327093127s) [noise=true , address=45.91.116.6:4975 , piece_id=SBVH62XUQOZI5SUS2AYNBAF2Z3G2IQ2OOXHFTVS5C2VVBPLXW44Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.301046413 (373h7m45.340619543s) [noise=true , address=77.90.31.12:4592 , piece_id=ZQO2KAY3ZL3WZCTZCJZNIRRJGDVXIFJCPNPAEENXUA6RD56GIPVQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:09.342349692 (373h7m45.381922812s) [noise=true , address=45.91.116.4:4937 , piece_id=4SVXWQCC4KWND6EMSYJ7NNMDKPGSOPSOLBN7LWKU6S4WLT2LWRMA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: <nil> Sep 22 02:36:36.120746520 (373h8m12.16031964s) [noise=true , address=145.79.227.12:4765 , piece_id=UQ7LFMRL7BQJALPLBIH3X22O3OQFFB4N34QEJTDE6GBHTDMJEHGA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:36.120818356 (373h8m12.160391486s) [noise=true , address=145.79.227.14:4839 , piece_id=R7D4P52YAFX5TKW35JVUZ6UN5ELCCS6FT6IVXW4K22CSK66NKCSA, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120876898 (373h8m12.160450018s) [noise=true , address=145.79.227.17:4970 , piece_id=BEEKRZFKETPRRMXTPDH6QLAIAQJAQQYMOFPBAFQIES7U7HNMMM7Q, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120944036 (373h8m12.160517156s) [noise=true , address=145.79.227.14:4838 , piece_id=6XPUIQJF7ELOYH7ZNBCU2SZL6EEH3WUP53LX76T3F6FEHYSRTWXQ, seg_pos={0 0}, upload=327956]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:36.120983501 (373h8m12.160556621s) [seg_pos={0 0}, upload=327956]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:36:36.368177143 (373h8m12.407750263s) [upload=327956]: Done uploading segments... Sep 22 02:36:36.368209475 (373h8m12.407782595s) [upload=327956]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.249370246s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........................XX........X.........X.....XXX.X.X.] duration: 6.084959094s FAIL (2025-09-22 02:36:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.084990963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...................XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...XX] duration: 30.249206449s FAIL (2025-09-22 02:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 02:35:36.352335007 (373h7m12.391908127s) [upload=327948]: Starting upload Sep 22 02:35:36.352392366 (373h7m12.391965496s) [upload=327948]: Done starting upload Sep 22 02:35:36.352491205 (373h7m12.392064325s) [upload=327948]: Uploading segments... Sep 22 02:35:36.352668542 (373h7m12.392241672s) [upload=327948]: Got next segment. Inline: false Sep 22 02:35:36.352725159 (373h7m12.392298279s) [upload=327948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:35:36.352782289 (373h7m12.392355419s) [upload=327948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:35:36.606053253 (373h7m12.645626383s) [seg_pos={0 0}, upload=327948]: Begin upload segment... Sep 22 02:35:36.606132574 (373h7m12.645705704s) [seg_pos={0 0}, upload=327948]: Done begin upload segment. Sep 22 02:35:36.606502548 (373h7m12.646075678s) [noise=true , address=45.91.116.9:4261 , piece_id=LNRTV77MN32WADMSUB7SSHZWI367SSUFIVZLMCNWKRTIVFBAO6TA, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.606653456 (373h7m12.646226596s) [noise=true , address=77.90.31.13:4628 , piece_id=NUPBUQWFAAXITGFNL5GGQKQFAJ7HIWOHQXXMJPG4JKWUYORIGM5A, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.607190007 (373h7m12.646763138s) [noise=true , address=45.91.116.4:4940 , piece_id=4KALPKZ75GAPW5XGHJADYAS4ENPSXIFOOUU6N32VJCYL7GSW2FSA, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.607350113 (373h7m12.646923243s) [noise=true , address=45.91.116.11:4118 , piece_id=HTK237AOMRVCEHSNRYQMIGBTSMBETO4SW4TJ4MQK2ILVW7FUI4JQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.608215731 (373h7m12.647788861s) [noise=true , address=45.91.116.8:4045 , piece_id=WUZ4BAW4EERXPJGMAV4MQTOJD5CXKHWXUUMCYVNLYXOACT3UD46Q, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.609127227 (373h7m12.648700357s) [noise=true , address=45.91.116.6:4975 , piece_id=YSD5GADGNTFYTLL5RWLG5HFYZRXUI65MHRIPJMAEWSBNGCCJ7IXA, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.610078287 (373h7m12.649651417s) [noise=true , address=45.91.116.14:4191 , piece_id=7S2WMEQK5RAQK5SOSYJ65NR3DQ2WH5PPDBR2IAWNULZASBV6LZMQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.610454614 (373h7m12.650027744s) [noise=true , address=145.79.227.16:4910 , piece_id=NMPFZIUQIM6ZATNKC2TVHT67UIR5EOWNK7Q43RWRC5MFPQRIG4FQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.610871818 (373h7m12.650444939s) [noise=true , address=145.79.227.12:4765 , piece_id=PSNUFPKZ62Y4H5MCT35LGAFY4CHE5REKEDCJQ4OIZSFYWIFMUPOA, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.612495330 (373h7m12.65206846s) [noise=true , address=145.79.227.12:4767 , piece_id=MBCHQEKM36VYME565OTATRTQMSKNZRKXASDWHO5SGGNWKMRVZUBA, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.614361263 (373h7m12.653934393s) [noise=true , address=45.91.116.7:4009 , piece_id=6TLJR2EFZZU7OVDIOQ6XVE5ADTBN2KPMNBEPEUAT2TT372KUREZQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.615331350 (373h7m12.65490448s) [noise=true , address=45.91.116.4:4937 , piece_id=WVFBVKSA6G5R3RKYAHPWNVZZ5QHQLGAIIWLFV233EVAUUHQX2SQQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.616306467 (373h7m12.655879587s) [noise=true , address=145.79.227.17:4971 , piece_id=TMJHE3ZDZNTIHG2TKOYK3C7XMF6EN46KPYXWOQGISKVADUAI3MSA, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.617936801 (373h7m12.657509932s) [noise=true , address=77.90.31.14:4663 , piece_id=DW6O4Z5CICOK6CCFTEN5Y4QFEORRUZIW57SYNKHGWOB5MJHJ2TAQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.618456581 (373h7m12.658029701s) [noise=true , address=45.91.116.10:4081 , piece_id=VXHHZRMTPEMNVGXNGP7NARSWEPD5CPAE776D63E4TOVX32E6WJTQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:36.618946745 (373h7m12.658519875s) [noise=true , address=145.79.227.15:4873 , piece_id=QCWUOQ65MKWX5UOGSI7VW5XYF32UOIC5QSBDF5PUX2WPNH7TOSHQ, seg_pos={0 0}, upload=327948]: Uploading piece... Sep 22 02:35:38.393684815 (373h7m14.433257945s) [upload=327948]: Next returned nil segment Sep 22 02:35:38.393708119 (373h7m14.433281239s) [upload=327948]: Waiting for error group managing segments... Sep 22 02:35:38.676135293 (373h7m14.715708413s) [noise=true , address=45.91.116.9:4261 , piece_id=LNRTV77MN32WADMSUB7SSHZWI367SSUFIVZLMCNWKRTIVFBAO6TA, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.689649937 (373h7m14.729223067s) [noise=true , address=45.91.116.10:4081 , piece_id=VXHHZRMTPEMNVGXNGP7NARSWEPD5CPAE776D63E4TOVX32E6WJTQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.706642805 (373h7m14.746215926s) [noise=true , address=45.91.116.7:4009 , piece_id=6TLJR2EFZZU7OVDIOQ6XVE5ADTBN2KPMNBEPEUAT2TT372KUREZQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.715329105 (373h7m14.754902235s) [noise=true , address=45.91.116.4:4940 , piece_id=4KALPKZ75GAPW5XGHJADYAS4ENPSXIFOOUU6N32VJCYL7GSW2FSA, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.738119212 (373h7m14.777692342s) [noise=true , address=45.91.116.8:4045 , piece_id=WUZ4BAW4EERXPJGMAV4MQTOJD5CXKHWXUUMCYVNLYXOACT3UD46Q, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.749113909 (373h7m14.788687039s) [noise=true , address=45.91.116.4:4937 , piece_id=WVFBVKSA6G5R3RKYAHPWNVZZ5QHQLGAIIWLFV233EVAUUHQX2SQQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.761269745 (373h7m14.800842875s) [noise=true , address=45.91.116.11:4118 , piece_id=HTK237AOMRVCEHSNRYQMIGBTSMBETO4SW4TJ4MQK2ILVW7FUI4JQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.792779124 (373h7m14.832352255s) [noise=true , address=77.90.31.13:4628 , piece_id=NUPBUQWFAAXITGFNL5GGQKQFAJ7HIWOHQXXMJPG4JKWUYORIGM5A, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.804085173 (373h7m14.843658303s) [noise=true , address=77.90.31.14:4663 , piece_id=DW6O4Z5CICOK6CCFTEN5Y4QFEORRUZIW57SYNKHGWOB5MJHJ2TAQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.831303883 (373h7m14.870877003s) [noise=true , address=45.91.116.14:4191 , piece_id=7S2WMEQK5RAQK5SOSYJ65NR3DQ2WH5PPDBR2IAWNULZASBV6LZMQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:35:38.852557984 (373h7m14.892131115s) [noise=true , address=45.91.116.6:4975 , piece_id=YSD5GADGNTFYTLL5RWLG5HFYZRXUI65MHRIPJMAEWSBNGCCJ7IXA, seg_pos={0 0}, upload=327948]: Done uploading piece. err: <nil> Sep 22 02:36:05.644909350 (373h7m41.68448248s) [noise=true , address=145.79.227.17:4971 , piece_id=TMJHE3ZDZNTIHG2TKOYK3C7XMF6EN46KPYXWOQGISKVADUAI3MSA, seg_pos={0 0}, upload=327948]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:05.644957130 (373h7m41.68453026s) [noise=true , address=145.79.227.12:4765 , piece_id=PSNUFPKZ62Y4H5MCT35LGAFY4CHE5REKEDCJQ4OIZSFYWIFMUPOA, seg_pos={0 0}, upload=327948]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:05.645073042 (373h7m41.684646172s) [noise=true , address=145.79.227.16:4910 , piece_id=NMPFZIUQIM6ZATNKC2TVHT67UIR5EOWNK7Q43RWRC5MFPQRIG4FQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:36:05.645108259 (373h7m41.684681399s) [noise=true , address=145.79.227.12:4767 , piece_id=MBCHQEKM36VYME565OTATRTQMSKNZRKXASDWHO5SGGNWKMRVZUBA, seg_pos={0 0}, upload=327948]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:05.645207688 (373h7m41.684780818s) [noise=true , address=145.79.227.15:4873 , piece_id=QCWUOQ65MKWX5UOGSI7VW5XYF32UOIC5QSBDF5PUX2WPNH7TOSHQ, seg_pos={0 0}, upload=327948]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:36:05.645231352 (373h7m41.684804482s) [seg_pos={0 0}, upload=327948]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 02:36:05.891444369 (373h7m41.931017499s) [upload=327948]: Done uploading segments... Sep 22 02:36:05.891480257 (373h7m41.931053387s) [upload=327948]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.249234061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................XXXXX...XX.............................] duration: 6.162989506s FAIL (2025-09-22 02:35:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.163026176s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................XX........X.........X.....XXX.X.X] duration: 30.00193167s FAIL (2025-09-22 02:35:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001957028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....................XX...XX.XXX.XXX..X..XXX.XX.XX..X.X.X...X] duration: 30.2491622s FAIL (2025-09-22 02:35:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:34:37.982084069 (373h6m14.021657199s) [upload=327934]: Starting upload Sep 22 02:34:37.982200811 (373h6m14.021773931s) [upload=327934]: Done starting upload Sep 22 02:34:37.982354885 (373h6m14.021928005s) [upload=327934]: Uploading segments... Sep 22 02:34:37.982496354 (373h6m14.022069484s) [upload=327934]: Got next segment. Inline: false Sep 22 02:34:37.982545638 (373h6m14.022118768s) [upload=327934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:34:37.982551779 (373h6m14.022124899s) [upload=327934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:34:38.237612964 (373h6m14.277186114s) [seg_pos={0 0}, upload=327934]: Begin upload segment... Sep 22 02:34:38.237690982 (373h6m14.277264122s) [seg_pos={0 0}, upload=327934]: Done begin upload segment. Sep 22 02:34:38.237707894 (373h6m14.277281024s) [noise=true , address=145.79.227.15:4873 , piece_id=Y5HCGBR7ST65NZR6TVUVVKC7GGZNGNBIGSBCXB7IMAJDTGEMOS3Q, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.237747149 (373h6m14.27732029s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDZ7OJXERPBFSJB5CRGH4FENOXSQMDW2B2PY4XSXQCK3YNVDP3Q, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.238286686 (373h6m14.277859806s) [noise=true , address=77.90.31.15:4699 , piece_id=OBB4TLG7SKX42IHBK4AJVTABTR3RVEQADFX353YH7VWPMUFSRDJA, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.238301093 (373h6m14.277874223s) [noise=true , address=45.91.116.12:4227 , piece_id=6EIKFTYJXC43C7UUHUVAFGU6Q6H7CRVO7FUYPTDOFSVEAOUV3UGQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.238672381 (373h6m14.278245501s) [noise=true , address=145.79.227.5:4221 , piece_id=TR2L6XEUDGM76KMJ3EYR4YINM3GASNCCJLGTUFGBYRI4D4WNEHPA, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.238726083 (373h6m14.278299213s) [noise=true , address=77.90.31.10:4517 , piece_id=QJLNSGARPXS5JJ6TULSZ7NOSRKPKBH3RL4EOCZKKZ53GLDV4LE2Q, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.239619674 (373h6m14.279192804s) [noise=true , address=45.91.116.8:4048 , piece_id=WF3H3HX3WM5M262DINORXTLCP2F7G6NGJWBX3NNSYQOBOLTAWEXQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.240341259 (373h6m14.279914389s) [noise=true , address=145.79.227.2:4114 , piece_id=MDAJGPVXPM6KBRPLQ26CAOYENRWGF5JNZE5LVOUXHTWZGUNHUTNQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.240964265 (373h6m14.280537425s) [noise=true , address=45.91.116.9:4263 , piece_id=3KQBPSXACXVQEXDI4YCJG7HTQTIYFBQRLPBBZ65ECYUHU2DW6DQQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.241683876 (373h6m14.281257016s) [noise=true , address=77.90.31.18:4807 , piece_id=J5QNAQBAMH4X2THOHB2MBIVMRRTDGSJFFS3EEWNNYJ2VWX2YBRTQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.241848639 (373h6m14.281421759s) [noise=true , address=77.90.31.18:4805 , piece_id=FFKONPOUFPZYJ2CA4FDSKMYAGBXJXVAYXVPN2NQPMRCMGC5LXZZA, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.242839837 (373h6m14.282412967s) [noise=true , address=45.91.116.8:4047 , piece_id=SQ3DQO2ZQBESLU63E5MCC57PLJHJ54EVMLF5FTYTLFB57KMU2EKQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.243939360 (373h6m14.28351253s) [noise=true , address=45.91.116.6:4975 , piece_id=VVSYPVDGXEI6YXXVJJCPN2255Y7GGKYGIYJJTBDYNND3IDJLXLMQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.244679720 (373h6m14.28425285s) [noise=true , address=77.90.31.10:4520 , piece_id=ZDMFUOI3BOPRCVTBS2M6GJ2LEX2QFKMWKWEEVNAM3GVBA4SEB5TQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.245653164 (373h6m14.285226304s) [noise=true , address=45.91.116.7:4009 , piece_id=J6EM5RDNVNDWBHQSH5B3E5OXLRMCWUC26OR7J7LIQLTS3ADNQSVQ, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:38.246285397 (373h6m14.285858528s) [noise=true , address=145.79.227.18:4007 , piece_id=ZKV24UZQDZ3Q3OXYPUBYTLRQETWMMHWSHXJKEPXRDWD4GVZMNOGA, seg_pos={0 0}, upload=327934]: Uploading piece... Sep 22 02:34:39.937043524 (373h6m15.976616644s) [upload=327934]: Next returned nil segment Sep 22 02:34:39.937062841 (373h6m15.976635961s) [upload=327934]: Waiting for error group managing segments... Sep 22 02:34:40.323492836 (373h6m16.363065966s) [noise=true , address=45.91.116.7:4009 , piece_id=J6EM5RDNVNDWBHQSH5B3E5OXLRMCWUC26OR7J7LIQLTS3ADNQSVQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.341047653 (373h6m16.380620773s) [noise=true , address=77.90.31.15:4699 , piece_id=OBB4TLG7SKX42IHBK4AJVTABTR3RVEQADFX353YH7VWPMUFSRDJA, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.342356175 (373h6m16.381929305s) [noise=true , address=45.91.116.8:4048 , piece_id=WF3H3HX3WM5M262DINORXTLCP2F7G6NGJWBX3NNSYQOBOLTAWEXQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.348898472 (373h6m16.388471602s) [noise=true , address=45.91.116.6:4975 , piece_id=VVSYPVDGXEI6YXXVJJCPN2255Y7GGKYGIYJJTBDYNND3IDJLXLMQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.402438914 (373h6m16.442012034s) [noise=true , address=45.91.116.12:4227 , piece_id=6EIKFTYJXC43C7UUHUVAFGU6Q6H7CRVO7FUYPTDOFSVEAOUV3UGQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.409000518 (373h6m16.448573638s) [noise=true , address=77.90.31.10:4520 , piece_id=ZDMFUOI3BOPRCVTBS2M6GJ2LEX2QFKMWKWEEVNAM3GVBA4SEB5TQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.428758320 (373h6m16.46833144s) [noise=true , address=45.91.116.9:4263 , piece_id=3KQBPSXACXVQEXDI4YCJG7HTQTIYFBQRLPBBZ65ECYUHU2DW6DQQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.431652331 (373h6m16.471225461s) [noise=true , address=77.90.31.18:4807 , piece_id=J5QNAQBAMH4X2THOHB2MBIVMRRTDGSJFFS3EEWNNYJ2VWX2YBRTQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.447878780 (373h6m16.48745191s) [noise=true , address=45.91.116.8:4047 , piece_id=SQ3DQO2ZQBESLU63E5MCC57PLJHJ54EVMLF5FTYTLFB57KMU2EKQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.467613087 (373h6m16.507186217s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDZ7OJXERPBFSJB5CRGH4FENOXSQMDW2B2PY4XSXQCK3YNVDP3Q, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.490916090 (373h6m16.53048922s) [noise=true , address=77.90.31.10:4517 , piece_id=QJLNSGARPXS5JJ6TULSZ7NOSRKPKBH3RL4EOCZKKZ53GLDV4LE2Q, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:34:40.526132288 (373h6m16.565705418s) [noise=true , address=77.90.31.18:4805 , piece_id=FFKONPOUFPZYJ2CA4FDSKMYAGBXJXVAYXVPN2NQPMRCMGC5LXZZA, seg_pos={0 0}, upload=327934]: Done uploading piece. err: <nil> Sep 22 02:35:07.270069289 (373h6m43.309642409s) [noise=true , address=145.79.227.2:4114 , piece_id=MDAJGPVXPM6KBRPLQ26CAOYENRWGF5JNZE5LVOUXHTWZGUNHUTNQ, seg_pos={0 0}, upload=327934]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: rpc: dial tcp 145.79.227.2:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:35:07.270091462 (373h6m43.309664592s) [noise=true , address=145.79.227.15:4873 , piece_id=Y5HCGBR7ST65NZR6TVUVVKC7GGZNGNBIGSBCXB7IMAJDTGEMOS3Q, seg_pos={0 0}, upload=327934]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:35:07.270117953 (373h6m43.309691083s) [noise=true , address=145.79.227.5:4221 , piece_id=TR2L6XEUDGM76KMJ3EYR4YINM3GASNCCJLGTUFGBYRI4D4WNEHPA, seg_pos={0 0}, upload=327934]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:35:07.270147880 (373h6m43.30972101s) [noise=true , address=145.79.227.18:4007 , piece_id=ZKV24UZQDZ3Q3OXYPUBYTLRQETWMMHWSHXJKEPXRDWD4GVZMNOGA, seg_pos={0 0}, upload=327934]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:35:07.270189980 (373h6m43.30976311s) [seg_pos={0 0}, upload=327934]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:35:07.516006170 (373h6m43.55557931s) [upload=327934]: Done uploading segments... Sep 22 02:35:07.516043350 (373h6m43.55561648s) [upload=327934]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.249197937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................XXXXX...XX............................] duration: 5.883086518s FAIL (2025-09-22 02:34:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 45s) [upload=327663]: Done starting upload Sep 22 02:16:37.914545401 (372h48m13.954118531s) [upload=327663]: Uploading segments... Sep 22 02:16:37.914717719 (372h48m13.954290879s) [upload=327663]: Got next segment. Inline: false Sep 22 02:16:37.914778555 (372h48m13.954351685s) [upload=327663]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:16:37.914786280 (372h48m13.95435941s) [upload=327663]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:16:38.380159445 (372h48m14.419732575s) [seg_pos={0 0}, upload=327663]: Begin upload segment... Sep 22 02:16:38.380224709 (372h48m14.419797829s) [seg_pos={0 0}, upload=327663]: Done begin upload segment. Sep 22 02:16:38.380242764 (372h48m14.419815894s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380270748 (372h48m14.419843878s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.380805205 (372h48m14.420378325s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381174108 (372h48m14.420747228s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381547408 (372h48m14.421120538s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.381946859 (372h48m14.421519989s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382325059 (372h48m14.42189819s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.382696175 (372h48m14.422269306s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383049830 (372h48m14.42262296s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383408853 (372h48m14.422981983s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.383773418 (372h48m14.423346548s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384179911 (372h48m14.423753041s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384554925 (372h48m14.424128056s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.384938596 (372h48m14.424511726s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385309232 (372h48m14.424882362s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.385703352 (372h48m14.425276482s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386128582 (372h48m14.425701702s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:38.386464221 (372h48m14.426037341s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Uploading piece... Sep 22 02:16:39.710969425 (372h48m15.750542555s) [upload=327663]: Next returned nil segment Sep 22 02:16:39.710988742 (372h48m15.750561872s) [upload=327663]: Waiting for error group managing segments... Sep 22 02:16:40.078015439 (372h48m16.117588569s) [noise=true , address=66.23.193.175:4560 , piece_id=AL3OR4THYX46UTSGPCW6CNT5J5JA4B3PHGTOFMQYUZ22EQXBWXGQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.165347856 (372h48m16.204920986s) [noise=true , address=64.49.13.26:4951 , piece_id=LF5WWSPF2CR3DGCZXIYIVRBSUHGYYJ7BZTCSTGZZAK56QUW4ZUAA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.267020530 (372h48m16.30659366s) [noise=true , address=145.79.227.33:4881 , piece_id=XOQX223DJK3TU4RJJK2XDDD6TNJZCFYZPHBBZIXU2Q7YCKX2EZDQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.294336745 (372h48m16.333909875s) [noise=true , address=145.79.227.45:4251 , piece_id=FQE7D47DOFMBEK2KYPFPHL3JLD6D6NAF3KK55LZYNXEMADFMTURA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.295321640 (372h48m16.33489476s) [noise=true , address=145.79.227.46:4620 , piece_id=NXK5F5WVBYWRZITLLBW4HXIMTP3PBDOWDHFPFKEECRCX22KVWKIA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.303026873 (372h48m16.342599993s) [noise=true , address=145.79.227.24:4609 , piece_id=IVIU4QLF2ELDIZTISK7JXDECZF6F5KGIHWZSB2YSTXZTUURZVL6Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.305300402 (372h48m16.344873522s) [noise=true , address=145.79.227.23:4550 , piece_id=VW3D6VL26DP7ILYILS3EX7F7OXBPDIKYGKBYD6O55VY7W3YAURVQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.315789074 (372h48m16.355362204s) [noise=true , address=145.79.227.31:4806 , piece_id=WIYDLXYYUDRISFVHXIHBS5NHVWKL75JUT5IOG4BSSNQDDWYAKGFA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.328763781 (372h48m16.368336911s) [noise=true , address=145.79.227.36:4530 , piece_id=ORIHXMZRNS4EA43WF2OZUUWSV73Y2ASTE5N4NROY5J6DPUM7YLZQ, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345845887 (372h48m16.385419018s) [noise=true , address=77.90.31.10:4539 , piece_id=ISTWBD3PXE4DHSA7FJUYN3RFKNFLAEH5GJVYMPI5EFEUAZMJCDCA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.345932042 (372h48m16.385505172s) [noise=true , address=77.90.31.18:4820 , piece_id=PXX7OQ5FAJY5XZZB65JJWQC3VPOKB5KJ6QSDPPJB7FSCC523KIBA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.364239583 (372h48m16.403812713s) [noise=true , address=77.90.31.16:4767 , piece_id=MN5KGNZSBHL7CM3C6IODGUGAEAB5HITP2JZ2CN7KHPU2QJRGC75Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:16:40.377476579 (372h48m16.417049709s) [noise=true , address=77.90.31.14:4672 , piece_id=4MSGWE6I7MZ6RLD2W6B2SG7H5IFKBE7UNKCRA72GA7HEURHNQMMA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: <nil> Sep 22 02:17:37.231910084 (372h49m13.271483214s) [noise=true , address=145.79.227.15:4907 , piece_id=EBLVQENYLLMOHVEBBPDWAAH3LWKW3VK2QJ76L2LPY75MPBTXX67A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3, address:145.79.227.15:4907): protocol: rpc: dial tcp 145.79.227.15:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232102199 (372h49m13.271675319s) [noise=true , address=145.79.227.12:4781 , piece_id=MYV3TRVPINDYNDNGULKRNBMZRDVBFIFCTGEVC66XHQVTQTVGID4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:145.79.227.12:4781): protocol: rpc: dial tcp 145.79.227.12:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232178735 (372h49m13.271751865s) [noise=true , address=145.79.227.19:4108 , piece_id=TAPELS2DGAWT6ZEKZIUB3OJKGRCWCLRRKMIAUCXU7RH45BYP4ZJA, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:145.79.227.19:4108): protocol: rpc: dial tcp 145.79.227.19:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:37.232303072 (372h49m13.271876202s) [noise=true , address=145.79.227.8:4363 , piece_id=RGN7IJLM6KHTSQVQ54WHTDJQQHTXO5G5HGIHV5FAVHCS7TOKWT4Q, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1utxkAShN2ZyJH3ZXgAES2Ue1NYa1JGNkcJyzUGe4aRY3WL7Lg, address:145.79.227.8:4363): protocol: rpc: dial tcp 145.79.227.8:4363: i/o timeout; context deadline exceeded Sep 22 02:17:37.232342677 (372h49m13.271915808s) [noise=true , address=145.79.227.18:4025 , piece_id=NVSAX575BGHHVAT6J4I7BSCJU42QBWREXIRMZ6RPN3GAEGTKR55A, seg_pos={0 0}, upload=327663]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:145.79.227.18:4025): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:37.232380399 (372h49m13.271953529s) [seg_pos={0 0}, upload=327663]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:17:37.471378747 (372h49m13.510951867s) [upload=327663]: Done uploading segments... Sep 22 02:17:37.471410427 (372h49m13.510983547s) [upload=327663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.883198822s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)