Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.83334867s FAIL (2025-07-16 18:13:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.833384179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.718362799s FAIL (2025-07-16 18:13:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.718392165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.792046413s FAIL (2025-07-16 18:12:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.792074668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.807035678s FAIL (2025-07-16 18:11:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.807061548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.034492601s FAIL (2025-07-16 18:10:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 6.03451837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........X................................................] duration: 5.7299175s FAIL (2025-07-16 18:09:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.729943219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..........X...............................................] duration: 5.845502132s FAIL (2025-07-16 18:08:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.845529353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..........X..............................................] duration: 5.707619639s FAIL (2025-07-16 18:07:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.707643755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..........X.............................................] duration: 5.867672284s FAIL (2025-07-16 18:06:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.86769583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..........X............................................] duration: 5.864982201s FAIL (2025-07-16 18:06:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.865006888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........X...........................................] duration: 5.93212063s FAIL (2025-07-16 18:05:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.932144776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..........X..........................................] duration: 5.739146556s FAIL (2025-07-16 18:03:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.739177014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........X.........................................] duration: 5.872316339s FAIL (2025-07-16 18:03:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.872346827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..........X........................................] duration: 6.034607179s FAIL (2025-07-16 18:02:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 6.034637286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........X.......................................] duration: 6.087706961s FAIL (2025-07-16 18:02:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 6.087730816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........X......................................] duration: 6.115205652s FAIL (2025-07-16 18:01:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 6.115232354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........X.....................................] duration: 5.855532051s FAIL (2025-07-16 18:01:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.855558802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..........X....................................] duration: 5.858733415s FAIL (2025-07-16 18:00:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.858768732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..........X...................................] duration: 6.113694701s FAIL (2025-07-16 17:59:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 6.113721562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..........X..................................] duration: 5.867799092s FAIL (2025-07-16 17:58:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.867817149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........X.................................] duration: 5.799543498s FAIL (2025-07-16 17:58:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.799567815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..........X................................] duration: 5.890629431s FAIL (2025-07-16 17:57:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.890645963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X................X..........X...............................] duration: 5.843822286s FAIL (2025-07-16 17:56:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.843839221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X................X..........X..............................] duration: 5.735090096s FAIL (2025-07-16 17:56:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.73511874s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 911.371476ms target: https://us1.storj.io/login FAIL (2025-07-16 17:56:11 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 911.408728ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 840.747998ms target: https://us1.storj.io/login FAIL (2025-07-16 17:55:22 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 840.778346ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X................X..........X.............................] duration: 5.77603512s FAIL (2025-07-16 17:55:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.776068793s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 894.093577ms target: https://us1.storj.io/login FAIL (2025-07-16 17:54:20 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 894.113384ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X................X..........X............................] duration: 5.906901759s FAIL (2025-07-16 17:54:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.906927469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X................X..........X...........................] duration: 5.873462555s FAIL (2025-07-16 17:53:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.873487933s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 647.825518ms target: https://us1.storj.io/login FAIL (2025-07-16 17:53:16 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 647.851007ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X................X..........X..........................] duration: 18.745564178s FAIL (2025-07-16 17:52:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 18.745592302s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 638.632589ms target: https://us1.storj.io/login FAIL (2025-07-16 17:52:49 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 638.654491ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X................X..........X.........................] duration: 5.989087203s FAIL (2025-07-16 17:52:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.989115095s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 640.576489ms target: https://us1.storj.io/login FAIL (2025-07-16 17:52:09 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 640.605294ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X................X..........X........................] duration: 5.986909354s FAIL (2025-07-16 17:51:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.986941114s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 642.700302ms target: https://us1.storj.io/login FAIL (2025-07-16 17:51:30 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 642.74058ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X................X..........X.......................] duration: 5.750762764s FAIL (2025-07-16 17:50:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.750790418s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 642.627674ms target: https://us1.storj.io/login FAIL (2025-07-16 17:50:19 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 642.654063ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X................X..........X......................] duration: 5.841590268s FAIL (2025-07-16 17:49:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.841614104s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 836.250818ms target: https://us1.storj.io/login FAIL (2025-07-16 17:49:35 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 836.276487ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X................X..........X.....................] duration: 5.740473787s FAIL (2025-07-16 17:49:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.740501913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X................X..........X....................] duration: 5.800266383s FAIL (2025-07-16 17:49:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.800296641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X................X..........X...................] duration: 5.706600817s FAIL (2025-07-16 17:48:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.706628239s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 649.517867ms target: https://us1.storj.io/login FAIL (2025-07-16 17:48:27 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 649.544288ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X................X..........X..................] duration: 5.7609585s FAIL (2025-07-16 17:47:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.760986715s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 723.744096ms target: https://us1.storj.io/login FAIL (2025-07-16 17:47:51 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 723.770866ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X................X..........X.................] duration: 5.79367534s FAIL (2025-07-16 17:47:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.793700139s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 645.612033ms target: https://us1.storj.io/login FAIL (2025-07-16 17:46:51 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 645.640738ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X................X..........X................] duration: 5.831085351s FAIL (2025-07-16 17:46:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tem_SegmentCommit Jul 16 17:33:17.710374760 (560h22m52.481016095s) [upload=448118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 17:33:18.176441596 (560h22m52.947082961s) [upload=448118]: Done uploading segments... Jul 16 17:33:18.176464019 (560h22m52.947105364s) [upload=448118]: Upload finished. err: <nil> Jul 16 17:34:24.404990334 (560h23m59.175631669s) [upload=448131]: Starting upload Jul 16 17:34:24.405048284 (560h23m59.17568962s) [upload=448131]: Done starting upload Jul 16 17:34:24.405137725 (560h23m59.17577906s) [upload=448131]: Uploading segments... Jul 16 17:34:24.405225243 (560h23m59.175866578s) [upload=448131]: Got next segment. Inline: false Jul 16 17:34:24.405251323 (560h23m59.175892658s) [upload=448131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 17:34:24.405255751 (560h23m59.175897086s) [upload=448131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 17:34:24.746894352 (560h23m59.517535687s) [seg_pos={0 0}, upload=448131]: Begin upload segment... Jul 16 17:34:24.746940491 (560h23m59.517581816s) [seg_pos={0 0}, upload=448131]: Done begin upload segment. Jul 16 17:34:24.746967572 (560h23m59.517608908s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.746992390 (560h23m59.517633735s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747280310 (560h23m59.517921646s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747583881 (560h23m59.518225206s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.747873596 (560h23m59.518514931s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748056816 (560h23m59.518698151s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748429479 (560h23m59.519070814s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748770041 (560h23m59.519411376s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.748866055 (560h23m59.5195074s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749159557 (560h23m59.519800882s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749414144 (560h23m59.520055479s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749526007 (560h23m59.520167343s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749651809 (560h23m59.520293144s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.749903189 (560h23m59.520544524s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.750047255 (560h23m59.52068859s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:24.752454691 (560h23m59.523096026s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Uploading piece... Jul 16 17:34:26.444711640 (560h24m1.215352975s) [upload=448131]: Next returned nil segment Jul 16 17:34:26.444757939 (560h24m1.215399274s) [upload=448131]: Waiting for error group managing segments... Jul 16 17:34:26.792535617 (560h24m1.563176952s) [noise=true , address=104.36.82.8:4972 , piece_id=32G7DXHJC4BC4AVFDHQ2AGIUAKVAU3MRKRGHV43OAHBFUSVFOGEQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.794159945 (560h24m1.56480128s) [noise=true , address=104.36.82.13:4185 , piece_id=7DDQHMP63IZ7VKTP6AWSURELFA7TZOPRM374IEREBICZ52SQK7IQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.810601202 (560h24m1.581242537s) [noise=true , address=104.36.82.3:4767 , piece_id=LPM2KMZNPVVJ2OBAB2AJD3SXI3QQ54YQHO7FZTXY64OISUMF3RLA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.812727070 (560h24m1.583368405s) [noise=true , address=104.36.82.6:4873 , piece_id=PA5Q5EYREVNVS4BOKJQRHECNNRXAHOYAXMBNQNXQKT7U42FMAJXQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.881085142 (560h24m1.651726477s) [noise=true , address=104.36.82.16:4294 , piece_id=RWB2PZDI2ID6SPD2HUSIRZYR4VSYNVQ2CCARGSH5WY5BXWYONUKQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.895742627 (560h24m1.666383963s) [noise=true , address=104.36.82.11:4115 , piece_id=OX4MPFH372FGFSYOED63JXZDTJNCX3CZLGUEAI7EMDBUP2DCZVZA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.953858663 (560h24m1.724499998s) [noise=true , address=104.36.82.5:4837 , piece_id=A3U2EB5QKVWKGPWDDBN7OEFAW2BISATXR7JSYNJMOVIABH5KW6SA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.958737709 (560h24m1.729379034s) [noise=true , address=104.36.82.10:4077 , piece_id=CPNT7XOOQT4GMPYUL4KS7HUNLF4NB42HVT7SPERN3QPZUWYGSOSA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.962052380 (560h24m1.732693715s) [noise=true , address=104.36.82.9:4007 , piece_id=ZU7MVTDTKSLFQNDFZBIY42YF3KUVDWSZKB6VDKZKM6XOL3JMLZRA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.966448942 (560h24m1.737090267s) [noise=true , address=104.36.82.2:4732 , piece_id=GEVJ235LSMKPK2I46CZZYDJWUJTYOOIK7QSLFDPMP37ABOMDJJDA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:26.996082532 (560h24m1.766723877s) [noise=true , address=104.36.82.11:4114 , piece_id=J52V6D3YSK3TVZDYUFEF2VAEKZ35QSIN7GWVA7LRW6L2T2WFEJTA, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:27.067684761 (560h24m1.838326146s) [noise=true , address=45.91.116.5:4299 , piece_id=MBGWZOZSY63LLV4DK7VHEIQVFJ4CS6FF3B7T2T6CQSPK7HDXY76A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: <nil> Jul 16 17:34:53.765713115 (560h24m28.536354451s) [noise=true , address=77.90.31.12:4592 , piece_id=BE3EGFPVK7OE4FOFKWRXEGGT3MHM7L3F47XMDB5XGHS5SEPPWCIQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42636->77.90.31.12:4592: use of closed network connection Jul 16 17:34:53.765770917 (560h24m28.536412252s) [noise=true , address=77.90.31.15:4699 , piece_id=M7GQWLBVZHPRB2VVKQSAVBB7A2LGVB46XRLBPLF27NJIXL4Z5QMQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35344->77.90.31.15:4699: use of closed network connection Jul 16 17:34:53.765782188 (560h24m28.536423523s) [noise=true , address=77.90.31.19:4842 , piece_id=EMWJAA6GZKVG2CZ2CNLHUBEJYBH752ER6RXBVTEAJL44ZMH4D7AQ, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:59270->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 17:34:53.765860638 (560h24m28.536501973s) [noise=true , address=77.90.31.11:4553 , piece_id=YI3MPUWTW7BYGYR3SZUYFIM2USBY367YV5CNZYM47E2Z7V6HDT3A, seg_pos={0 0}, upload=448131]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->77.90.31.11:4553: use of closed network connection Jul 16 17:34:53.765891447 (560h24m28.536532782s) [seg_pos={0 0}, upload=448131]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 17:34:53.994663536 (560h24m28.765304941s) [upload=448131]: Done uploading segments... Jul 16 17:34:53.994695617 (560h24m28.765336952s) [upload=448131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.831109597s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)