Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 6.001142093s FAIL (2025-07-16 09:27:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.001165146s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X........X.................] duration: 5.748386982s FAIL (2025-07-16 09:27:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.74845957s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 6.023408452s FAIL (2025-07-16 09:26:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.023429933s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X........X................] duration: 5.715018477s FAIL (2025-07-16 09:26:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.715044556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X........X...............] duration: 5.725476748s FAIL (2025-07-16 09:26:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.725500915s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 6.339620701s FAIL (2025-07-16 09:25:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.339650658s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X........X..............] duration: 5.714316875s FAIL (2025-07-16 09:25:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.714352884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X........X.............] duration: 5.920402339s FAIL (2025-07-16 09:24:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.920422305s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 6.300481117s FAIL (2025-07-16 09:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.300507828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X........X............] duration: 5.705904551s FAIL (2025-07-16 09:24:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.705926012s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 6.145285991s FAIL (2025-07-16 09:24:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.145488139s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 6.214583978s FAIL (2025-07-16 09:23:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.214606441s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X........X...........] duration: 5.710942066s FAIL (2025-07-16 09:23:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.710967095s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 6.153951792s FAIL (2025-07-16 09:23:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.153997219s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X........X..........] duration: 5.679967849s FAIL (2025-07-16 09:22:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.67999409s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 5.846051038s FAIL (2025-07-16 09:22:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.846075185s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration: 5.818104685s FAIL (2025-07-16 09:21:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.818133309s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 5.922525737s FAIL (2025-07-16 09:21:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.922675594s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 6.389308065s FAIL (2025-07-16 09:21:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.389364844s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X........X........] duration: 5.901752068s FAIL (2025-07-16 09:20:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.90177947s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 15.117896882s FAIL (2025-07-16 09:20:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.117916349s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration: 5.739237028s FAIL (2025-07-16 09:19:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.739264831s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 6.302041708s FAIL (2025-07-16 09:19:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.302061697s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X........X......] duration: 5.751786745s FAIL (2025-07-16 09:18:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.751808237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 6.134226641s FAIL (2025-07-16 09:18:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.134248813s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration: 5.762543239s FAIL (2025-07-16 09:17:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.762567417s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 6.514855194s FAIL (2025-07-16 09:17:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.514879279s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X........X....] duration: 5.722520555s FAIL (2025-07-16 09:17:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.722546365s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....................................................X...X.] duration: 9.065178178s FAIL (2025-07-16 09:16:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.065340239s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X........X...] duration: 5.732808277s FAIL (2025-07-16 09:16:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.732831893s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....................................................X...X] duration: 30.000460843s FAIL (2025-07-16 09:16:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000482465s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X........X..] duration: 5.873892446s FAIL (2025-07-16 09:16:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.873918956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 28.370380469s FAIL (2025-07-16 09:16:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 28.370408953s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....................................................X...] duration: 6.131133302s FAIL (2025-07-16 09:15:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.13114815s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 30.24851553s FAIL (2025-07-16 09:14:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 30.248540338s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....................................................X..] duration: 6.323873916s FAIL (2025-07-16 09:14:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.323896959s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.758438815s FAIL (2025-07-16 09:14:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 09:06:05.868769252 (551h55m40.639410587s) [upload=441390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:06:06.222644751 (551h55m40.993286086s) [upload=441390]: Done uploading segments... Jul 16 09:06:06.222664297 (551h55m40.993305622s) [upload=441390]: Upload finished. err: <nil> Jul 16 09:06:54.735377570 (551h56m29.506018905s) [upload=441403]: Starting upload Jul 16 09:06:54.735438356 (551h56m29.506079691s) [upload=441403]: Done starting upload Jul 16 09:06:54.735553486 (551h56m29.506194821s) [upload=441403]: Uploading segments... Jul 16 09:06:54.735647496 (551h56m29.506288831s) [upload=441403]: Got next segment. Inline: false Jul 16 09:06:54.735691830 (551h56m29.506333156s) [upload=441403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:06:54.735696801 (551h56m29.506338126s) [upload=441403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:06:55.000517148 (551h56m29.771158493s) [seg_pos={0 0}, upload=441403]: Begin upload segment... Jul 16 09:06:55.000560552 (551h56m29.771201887s) [seg_pos={0 0}, upload=441403]: Done begin upload segment. Jul 16 09:06:55.000582484 (551h56m29.771223819s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.000938205 (551h56m29.77157955s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001333160 (551h56m29.771974485s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001368678 (551h56m29.772010063s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001588228 (551h56m29.772229563s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001852313 (551h56m29.772493638s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002219245 (551h56m29.77286058s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002555720 (551h56m29.773197055s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002908554 (551h56m29.773549889s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003159694 (551h56m29.773801029s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003392009 (551h56m29.774033344s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003645254 (551h56m29.774286589s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003869282 (551h56m29.774510617s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004111546 (551h56m29.774752881s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004402954 (551h56m29.775044289s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004663842 (551h56m29.775305178s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.256887489 (551h56m30.027528824s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:45420->45.91.116.5:4298: write: connection refused; conn closed: op Jul 16 09:06:56.766451722 (551h56m31.537093057s) [upload=441403]: Next returned nil segment Jul 16 09:06:56.766466841 (551h56m31.537108176s) [upload=441403]: Waiting for error group managing segments... Jul 16 09:06:57.054548646 (551h56m31.825189981s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.065564764 (551h56m31.836206099s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.066516524 (551h56m31.837157859s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.071875808 (551h56m31.842517143s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.074198142 (551h56m31.844839477s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.152436969 (551h56m31.923078304s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.195405372 (551h56m31.966046707s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.292185534 (551h56m32.06282688s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:14.395388762 (551h56m49.166030097s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:16.514614818 (551h56m51.285256153s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:19.662997817 (551h56m54.433639153s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:22.031751551 (551h56m56.802392886s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:24.082480631 (551h56m58.853121966s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection Jul 16 09:07:24.082556446 (551h56m58.853197781s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection Jul 16 09:07:24.082586183 (551h56m58.853227508s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection Jul 16 09:07:24.082607393 (551h56m58.853248728s) [seg_pos={0 0}, upload=441403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:07:24.317015634 (551h56m59.087656969s) [upload=441403]: Done uploading segments... Jul 16 09:07:24.317040031 (551h56m59.087681356s) [upload=441403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.758461378s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....................................................X.] duration: 13.586697571s FAIL (2025-07-16 09:14:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.586719784s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....................................................X] duration: 30.001190752s FAIL (2025-07-16 09:13:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001211011s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.952219098s FAIL (2025-07-16 09:13:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 09:06:05.868769252 (551h55m40.639410587s) [upload=441390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:06:06.222644751 (551h55m40.993286086s) [upload=441390]: Done uploading segments... Jul 16 09:06:06.222664297 (551h55m40.993305622s) [upload=441390]: Upload finished. err: <nil> Jul 16 09:06:54.735377570 (551h56m29.506018905s) [upload=441403]: Starting upload Jul 16 09:06:54.735438356 (551h56m29.506079691s) [upload=441403]: Done starting upload Jul 16 09:06:54.735553486 (551h56m29.506194821s) [upload=441403]: Uploading segments... Jul 16 09:06:54.735647496 (551h56m29.506288831s) [upload=441403]: Got next segment. Inline: false Jul 16 09:06:54.735691830 (551h56m29.506333156s) [upload=441403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:06:54.735696801 (551h56m29.506338126s) [upload=441403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:06:55.000517148 (551h56m29.771158493s) [seg_pos={0 0}, upload=441403]: Begin upload segment... Jul 16 09:06:55.000560552 (551h56m29.771201887s) [seg_pos={0 0}, upload=441403]: Done begin upload segment. Jul 16 09:06:55.000582484 (551h56m29.771223819s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.000938205 (551h56m29.77157955s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001333160 (551h56m29.771974485s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001368678 (551h56m29.772010063s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001588228 (551h56m29.772229563s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001852313 (551h56m29.772493638s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002219245 (551h56m29.77286058s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002555720 (551h56m29.773197055s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002908554 (551h56m29.773549889s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003159694 (551h56m29.773801029s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003392009 (551h56m29.774033344s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003645254 (551h56m29.774286589s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003869282 (551h56m29.774510617s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004111546 (551h56m29.774752881s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004402954 (551h56m29.775044289s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004663842 (551h56m29.775305178s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.256887489 (551h56m30.027528824s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:45420->45.91.116.5:4298: write: connection refused; conn closed: op Jul 16 09:06:56.766451722 (551h56m31.537093057s) [upload=441403]: Next returned nil segment Jul 16 09:06:56.766466841 (551h56m31.537108176s) [upload=441403]: Waiting for error group managing segments... Jul 16 09:06:57.054548646 (551h56m31.825189981s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.065564764 (551h56m31.836206099s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.066516524 (551h56m31.837157859s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.071875808 (551h56m31.842517143s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.074198142 (551h56m31.844839477s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.152436969 (551h56m31.923078304s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.195405372 (551h56m31.966046707s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.292185534 (551h56m32.06282688s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:14.395388762 (551h56m49.166030097s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:16.514614818 (551h56m51.285256153s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:19.662997817 (551h56m54.433639153s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:22.031751551 (551h56m56.802392886s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:24.082480631 (551h56m58.853121966s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection Jul 16 09:07:24.082556446 (551h56m58.853197781s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection Jul 16 09:07:24.082586183 (551h56m58.853227508s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection Jul 16 09:07:24.082607393 (551h56m58.853248728s) [seg_pos={0 0}, upload=441403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:07:24.317015634 (551h56m59.087656969s) [upload=441403]: Done uploading segments... Jul 16 09:07:24.317040031 (551h56m59.087681356s) [upload=441403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.952244546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.835265959s FAIL (2025-07-16 09:12:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 09:06:05.868769252 (551h55m40.639410587s) [upload=441390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:06:06.222644751 (551h55m40.993286086s) [upload=441390]: Done uploading segments... Jul 16 09:06:06.222664297 (551h55m40.993305622s) [upload=441390]: Upload finished. err: <nil> Jul 16 09:06:54.735377570 (551h56m29.506018905s) [upload=441403]: Starting upload Jul 16 09:06:54.735438356 (551h56m29.506079691s) [upload=441403]: Done starting upload Jul 16 09:06:54.735553486 (551h56m29.506194821s) [upload=441403]: Uploading segments... Jul 16 09:06:54.735647496 (551h56m29.506288831s) [upload=441403]: Got next segment. Inline: false Jul 16 09:06:54.735691830 (551h56m29.506333156s) [upload=441403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:06:54.735696801 (551h56m29.506338126s) [upload=441403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:06:55.000517148 (551h56m29.771158493s) [seg_pos={0 0}, upload=441403]: Begin upload segment... Jul 16 09:06:55.000560552 (551h56m29.771201887s) [seg_pos={0 0}, upload=441403]: Done begin upload segment. Jul 16 09:06:55.000582484 (551h56m29.771223819s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.000938205 (551h56m29.77157955s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001333160 (551h56m29.771974485s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001368678 (551h56m29.772010063s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001588228 (551h56m29.772229563s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001852313 (551h56m29.772493638s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002219245 (551h56m29.77286058s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002555720 (551h56m29.773197055s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002908554 (551h56m29.773549889s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003159694 (551h56m29.773801029s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003392009 (551h56m29.774033344s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003645254 (551h56m29.774286589s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003869282 (551h56m29.774510617s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004111546 (551h56m29.774752881s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004402954 (551h56m29.775044289s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004663842 (551h56m29.775305178s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.256887489 (551h56m30.027528824s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:45420->45.91.116.5:4298: write: connection refused; conn closed: op Jul 16 09:06:56.766451722 (551h56m31.537093057s) [upload=441403]: Next returned nil segment Jul 16 09:06:56.766466841 (551h56m31.537108176s) [upload=441403]: Waiting for error group managing segments... Jul 16 09:06:57.054548646 (551h56m31.825189981s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.065564764 (551h56m31.836206099s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.066516524 (551h56m31.837157859s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.071875808 (551h56m31.842517143s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.074198142 (551h56m31.844839477s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.152436969 (551h56m31.923078304s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.195405372 (551h56m31.966046707s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.292185534 (551h56m32.06282688s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:14.395388762 (551h56m49.166030097s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:16.514614818 (551h56m51.285256153s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:19.662997817 (551h56m54.433639153s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:22.031751551 (551h56m56.802392886s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:24.082480631 (551h56m58.853121966s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection Jul 16 09:07:24.082556446 (551h56m58.853197781s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection Jul 16 09:07:24.082586183 (551h56m58.853227508s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection Jul 16 09:07:24.082607393 (551h56m58.853248728s) [seg_pos={0 0}, upload=441403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:07:24.317015634 (551h56m59.087656969s) [upload=441403]: Done uploading segments... Jul 16 09:07:24.317040031 (551h56m59.087681356s) [upload=441403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.835293883s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.072479055s FAIL (2025-07-16 09:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.07250248s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.72885134s FAIL (2025-07-16 09:11:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 09:06:05.868769252 (551h55m40.639410587s) [upload=441390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:06:06.222644751 (551h55m40.993286086s) [upload=441390]: Done uploading segments... Jul 16 09:06:06.222664297 (551h55m40.993305622s) [upload=441390]: Upload finished. err: <nil> Jul 16 09:06:54.735377570 (551h56m29.506018905s) [upload=441403]: Starting upload Jul 16 09:06:54.735438356 (551h56m29.506079691s) [upload=441403]: Done starting upload Jul 16 09:06:54.735553486 (551h56m29.506194821s) [upload=441403]: Uploading segments... Jul 16 09:06:54.735647496 (551h56m29.506288831s) [upload=441403]: Got next segment. Inline: false Jul 16 09:06:54.735691830 (551h56m29.506333156s) [upload=441403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:06:54.735696801 (551h56m29.506338126s) [upload=441403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:06:55.000517148 (551h56m29.771158493s) [seg_pos={0 0}, upload=441403]: Begin upload segment... Jul 16 09:06:55.000560552 (551h56m29.771201887s) [seg_pos={0 0}, upload=441403]: Done begin upload segment. Jul 16 09:06:55.000582484 (551h56m29.771223819s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.000938205 (551h56m29.77157955s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001333160 (551h56m29.771974485s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001368678 (551h56m29.772010063s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001588228 (551h56m29.772229563s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001852313 (551h56m29.772493638s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002219245 (551h56m29.77286058s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002555720 (551h56m29.773197055s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002908554 (551h56m29.773549889s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003159694 (551h56m29.773801029s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003392009 (551h56m29.774033344s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003645254 (551h56m29.774286589s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003869282 (551h56m29.774510617s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004111546 (551h56m29.774752881s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004402954 (551h56m29.775044289s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004663842 (551h56m29.775305178s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.256887489 (551h56m30.027528824s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:45420->45.91.116.5:4298: write: connection refused; conn closed: op Jul 16 09:06:56.766451722 (551h56m31.537093057s) [upload=441403]: Next returned nil segment Jul 16 09:06:56.766466841 (551h56m31.537108176s) [upload=441403]: Waiting for error group managing segments... Jul 16 09:06:57.054548646 (551h56m31.825189981s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.065564764 (551h56m31.836206099s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.066516524 (551h56m31.837157859s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.071875808 (551h56m31.842517143s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.074198142 (551h56m31.844839477s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.152436969 (551h56m31.923078304s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.195405372 (551h56m31.966046707s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.292185534 (551h56m32.06282688s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:14.395388762 (551h56m49.166030097s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:16.514614818 (551h56m51.285256153s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:19.662997817 (551h56m54.433639153s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:22.031751551 (551h56m56.802392886s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:24.082480631 (551h56m58.853121966s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection Jul 16 09:07:24.082556446 (551h56m58.853197781s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection Jul 16 09:07:24.082586183 (551h56m58.853227508s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection Jul 16 09:07:24.082607393 (551h56m58.853248728s) [seg_pos={0 0}, upload=441403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:07:24.317015634 (551h56m59.087656969s) [upload=441403]: Done uploading segments... Jul 16 09:07:24.317040031 (551h56m59.087681356s) [upload=441403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.728890275s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.00918508s FAIL (2025-07-16 09:11:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.009205089s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.912182263s FAIL (2025-07-16 09:10:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.912198504s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.540726117s FAIL (2025-07-16 09:10:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.540739863s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.722338163s FAIL (2025-07-16 09:10:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 09:06:05.868769252 (551h55m40.639410587s) [upload=441390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:06:06.222644751 (551h55m40.993286086s) [upload=441390]: Done uploading segments... Jul 16 09:06:06.222664297 (551h55m40.993305622s) [upload=441390]: Upload finished. err: <nil> Jul 16 09:06:54.735377570 (551h56m29.506018905s) [upload=441403]: Starting upload Jul 16 09:06:54.735438356 (551h56m29.506079691s) [upload=441403]: Done starting upload Jul 16 09:06:54.735553486 (551h56m29.506194821s) [upload=441403]: Uploading segments... Jul 16 09:06:54.735647496 (551h56m29.506288831s) [upload=441403]: Got next segment. Inline: false Jul 16 09:06:54.735691830 (551h56m29.506333156s) [upload=441403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:06:54.735696801 (551h56m29.506338126s) [upload=441403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:06:55.000517148 (551h56m29.771158493s) [seg_pos={0 0}, upload=441403]: Begin upload segment... Jul 16 09:06:55.000560552 (551h56m29.771201887s) [seg_pos={0 0}, upload=441403]: Done begin upload segment. Jul 16 09:06:55.000582484 (551h56m29.771223819s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.000938205 (551h56m29.77157955s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001333160 (551h56m29.771974485s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001368678 (551h56m29.772010063s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001588228 (551h56m29.772229563s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001852313 (551h56m29.772493638s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002219245 (551h56m29.77286058s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002555720 (551h56m29.773197055s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002908554 (551h56m29.773549889s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003159694 (551h56m29.773801029s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003392009 (551h56m29.774033344s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003645254 (551h56m29.774286589s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003869282 (551h56m29.774510617s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004111546 (551h56m29.774752881s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004402954 (551h56m29.775044289s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004663842 (551h56m29.775305178s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.256887489 (551h56m30.027528824s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:45420->45.91.116.5:4298: write: connection refused; conn closed: op Jul 16 09:06:56.766451722 (551h56m31.537093057s) [upload=441403]: Next returned nil segment Jul 16 09:06:56.766466841 (551h56m31.537108176s) [upload=441403]: Waiting for error group managing segments... Jul 16 09:06:57.054548646 (551h56m31.825189981s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.065564764 (551h56m31.836206099s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.066516524 (551h56m31.837157859s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.071875808 (551h56m31.842517143s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.074198142 (551h56m31.844839477s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.152436969 (551h56m31.923078304s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.195405372 (551h56m31.966046707s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.292185534 (551h56m32.06282688s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:14.395388762 (551h56m49.166030097s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:16.514614818 (551h56m51.285256153s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:19.662997817 (551h56m54.433639153s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:22.031751551 (551h56m56.802392886s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:24.082480631 (551h56m58.853121966s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection Jul 16 09:07:24.082556446 (551h56m58.853197781s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection Jul 16 09:07:24.082586183 (551h56m58.853227508s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection Jul 16 09:07:24.082607393 (551h56m58.853248728s) [seg_pos={0 0}, upload=441403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:07:24.317015634 (551h56m59.087656969s) [upload=441403]: Done uploading segments... Jul 16 09:07:24.317040031 (551h56m59.087681356s) [upload=441403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.722366316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.720919127s FAIL (2025-07-16 09:09:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 09:06:05.868769252 (551h55m40.639410587s) [upload=441390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:06:06.222644751 (551h55m40.993286086s) [upload=441390]: Done uploading segments... Jul 16 09:06:06.222664297 (551h55m40.993305622s) [upload=441390]: Upload finished. err: <nil> Jul 16 09:06:54.735377570 (551h56m29.506018905s) [upload=441403]: Starting upload Jul 16 09:06:54.735438356 (551h56m29.506079691s) [upload=441403]: Done starting upload Jul 16 09:06:54.735553486 (551h56m29.506194821s) [upload=441403]: Uploading segments... Jul 16 09:06:54.735647496 (551h56m29.506288831s) [upload=441403]: Got next segment. Inline: false Jul 16 09:06:54.735691830 (551h56m29.506333156s) [upload=441403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:06:54.735696801 (551h56m29.506338126s) [upload=441403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:06:55.000517148 (551h56m29.771158493s) [seg_pos={0 0}, upload=441403]: Begin upload segment... Jul 16 09:06:55.000560552 (551h56m29.771201887s) [seg_pos={0 0}, upload=441403]: Done begin upload segment. Jul 16 09:06:55.000582484 (551h56m29.771223819s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.000938205 (551h56m29.77157955s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001333160 (551h56m29.771974485s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001368678 (551h56m29.772010063s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001588228 (551h56m29.772229563s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001852313 (551h56m29.772493638s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002219245 (551h56m29.77286058s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002555720 (551h56m29.773197055s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002908554 (551h56m29.773549889s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003159694 (551h56m29.773801029s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003392009 (551h56m29.774033344s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003645254 (551h56m29.774286589s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003869282 (551h56m29.774510617s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004111546 (551h56m29.774752881s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004402954 (551h56m29.775044289s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004663842 (551h56m29.775305178s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.256887489 (551h56m30.027528824s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:45420->45.91.116.5:4298: write: connection refused; conn closed: op Jul 16 09:06:56.766451722 (551h56m31.537093057s) [upload=441403]: Next returned nil segment Jul 16 09:06:56.766466841 (551h56m31.537108176s) [upload=441403]: Waiting for error group managing segments... Jul 16 09:06:57.054548646 (551h56m31.825189981s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.065564764 (551h56m31.836206099s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.066516524 (551h56m31.837157859s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.071875808 (551h56m31.842517143s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.074198142 (551h56m31.844839477s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.152436969 (551h56m31.923078304s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.195405372 (551h56m31.966046707s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.292185534 (551h56m32.06282688s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:14.395388762 (551h56m49.166030097s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:16.514614818 (551h56m51.285256153s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:19.662997817 (551h56m54.433639153s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:22.031751551 (551h56m56.802392886s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:24.082480631 (551h56m58.853121966s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection Jul 16 09:07:24.082556446 (551h56m58.853197781s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection Jul 16 09:07:24.082586183 (551h56m58.853227508s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection Jul 16 09:07:24.082607393 (551h56m58.853248728s) [seg_pos={0 0}, upload=441403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:07:24.317015634 (551h56m59.087656969s) [upload=441403]: Done uploading segments... Jul 16 09:07:24.317040031 (551h56m59.087681356s) [upload=441403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.720943783s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.435569618s FAIL (2025-07-16 09:09:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.435593184s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.993666605s FAIL (2025-07-16 09:08:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 09:06:05.868769252 (551h55m40.639410587s) [upload=441390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:06:06.222644751 (551h55m40.993286086s) [upload=441390]: Done uploading segments... Jul 16 09:06:06.222664297 (551h55m40.993305622s) [upload=441390]: Upload finished. err: <nil> Jul 16 09:06:54.735377570 (551h56m29.506018905s) [upload=441403]: Starting upload Jul 16 09:06:54.735438356 (551h56m29.506079691s) [upload=441403]: Done starting upload Jul 16 09:06:54.735553486 (551h56m29.506194821s) [upload=441403]: Uploading segments... Jul 16 09:06:54.735647496 (551h56m29.506288831s) [upload=441403]: Got next segment. Inline: false Jul 16 09:06:54.735691830 (551h56m29.506333156s) [upload=441403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:06:54.735696801 (551h56m29.506338126s) [upload=441403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:06:55.000517148 (551h56m29.771158493s) [seg_pos={0 0}, upload=441403]: Begin upload segment... Jul 16 09:06:55.000560552 (551h56m29.771201887s) [seg_pos={0 0}, upload=441403]: Done begin upload segment. Jul 16 09:06:55.000582484 (551h56m29.771223819s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.000938205 (551h56m29.77157955s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001333160 (551h56m29.771974485s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001368678 (551h56m29.772010063s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001588228 (551h56m29.772229563s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.001852313 (551h56m29.772493638s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002219245 (551h56m29.77286058s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002555720 (551h56m29.773197055s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.002908554 (551h56m29.773549889s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003159694 (551h56m29.773801029s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003392009 (551h56m29.774033344s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003645254 (551h56m29.774286589s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.003869282 (551h56m29.774510617s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004111546 (551h56m29.774752881s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004402954 (551h56m29.775044289s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.004663842 (551h56m29.775305178s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Uploading piece... Jul 16 09:06:55.256887489 (551h56m30.027528824s) [noise=true , address=45.91.116.5:4298 , piece_id=CGEWGLWQ7KWSGVYL5B2EBB4FGDHPMTGTD7GDFLETILIJNEPC3XCQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:45420->45.91.116.5:4298: write: connection refused; conn closed: op Jul 16 09:06:56.766451722 (551h56m31.537093057s) [upload=441403]: Next returned nil segment Jul 16 09:06:56.766466841 (551h56m31.537108176s) [upload=441403]: Waiting for error group managing segments... Jul 16 09:06:57.054548646 (551h56m31.825189981s) [noise=true , address=45.91.116.13:4154 , piece_id=CCHTPPIA6AM47XDHDGUU5SR7VWCYSRWA4PAKRQE452G3UXB3FJWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.065564764 (551h56m31.836206099s) [noise=true , address=45.91.116.4:4940 , piece_id=DVMXK6YKERBPJ5DW3XDLE5RIUB3IHK4SHFGEF45L6P3OXL5PEZ4Q, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.066516524 (551h56m31.837157859s) [noise=true , address=45.91.116.14:4189 , piece_id=TY2UFBVA52ZFOAQRYIUE3TFZFQTIZWPFAEV6QYNW3MAXV773R5VA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.071875808 (551h56m31.842517143s) [noise=true , address=45.91.116.11:4117 , piece_id=L4QT5LSC3YBVQCAS7AHUEOQHMIBMZL4HZL246QE7VXG4PSHZVHYA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.074198142 (551h56m31.844839477s) [noise=true , address=45.91.116.6:4975 , piece_id=WNIO2CADS265OVUXGYWRCEUTZPOSDCMWEJPUV7T5L4EHEWJVWOMQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.152436969 (551h56m31.923078304s) [noise=true , address=45.91.116.7:4010 , piece_id=VUCA5MC4E4VZTNYUTSOUE5QNAZO5S6P6U6PJYUEZZHIFGO3JR2TQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.195405372 (551h56m31.966046707s) [noise=true , address=104.36.82.12:4149 , piece_id=H6EMSIXA5ZLL2KZKQ5TMXPODHPIK6RAVR3WKOW4GPRW2XFC3RHOA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:06:57.292185534 (551h56m32.06282688s) [noise=true , address=104.36.82.5:4838 , piece_id=TTABSIZDVXSPCL547DSJVUU6JBHQ6HMUAFB6W3R3NZH6MLMFQ6CQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:14.395388762 (551h56m49.166030097s) [noise=true , address=77.90.31.16:4734 , piece_id=EE4F7GJ6ZBUAWYZWP4UH2XO46I7X5Y3ISAJ6DW7F22UPOYL44FDQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:16.514614818 (551h56m51.285256153s) [noise=true , address=77.90.31.19:4841 , piece_id=LWUUBUE4EYKBIR4BIJEW5T2ISKVH4PHGXKT5YT7ALCPBZS7CWOEQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:19.662997817 (551h56m54.433639153s) [noise=true , address=77.90.31.15:4697 , piece_id=64XIPM2CY3EVWRWMV4MLVVH4RTCIHNDNBEW7IYE2R4MFMQKLWPJQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:22.031751551 (551h56m56.802392886s) [noise=true , address=77.90.31.21:4879 , piece_id=MVMIEWUQQKJCUDO2I5ZEDFBPW54XZLHG5RJCL7SU7ULJJK6RXEHQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: <nil> Jul 16 09:07:24.082480631 (551h56m58.853121966s) [noise=true , address=77.90.31.15:4699 , piece_id=ODLQHB5SR2DQAZFWJ5BBQEDZHG63TTKI6LD4VUDT5HM57RCTOKTQ, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47992->77.90.31.15:4699: use of closed network connection Jul 16 09:07:24.082556446 (551h56m58.853197781s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWW3JDSK4ZPCO2MKAQEIYLIYJHTWIY2V5KLZACNMGLPLTGDD3FWA, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->77.90.31.14:4661: use of closed network connection Jul 16 09:07:24.082586183 (551h56m58.853227508s) [noise=true , address=77.90.31.18:4806 , piece_id=L6JQTVZ4ZXFUENLNUH24SBY26WM3FG2H5IPUFR3WIZJX7IOHT67A, seg_pos={0 0}, upload=441403]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46036->77.90.31.18:4806: use of closed network connection Jul 16 09:07:24.082607393 (551h56m58.853248728s) [seg_pos={0 0}, upload=441403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:07:24.317015634 (551h56m59.087656969s) [upload=441403]: Done uploading segments... Jul 16 09:07:24.317040031 (551h56m59.087681356s) [upload=441403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.993696763s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)