Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.803373872s FAIL (2025-06-26 06:21:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.803395254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.817093967s FAIL (2025-06-26 06:21:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.817113022s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX..............X......................X.....] duration: 5.952344695s FAIL (2025-06-26 06:21:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.952369092s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 5.563379782s FAIL (2025-06-26 06:21:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.563397817s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.703218976s FAIL (2025-06-26 06:20:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.70324188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX..............X......................X....] duration: 6.147076374s FAIL (2025-06-26 06:20:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.147095852s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 5.655220907s FAIL (2025-06-26 06:20:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.655248179s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.752190339s FAIL (2025-06-26 06:19:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.752210529s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX..............X......................X...] duration: 5.948027794s FAIL (2025-06-26 06:19:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.94805149s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.90194849s FAIL (2025-06-26 06:19:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.901964179s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XX..............X......................X..] duration: 6.081385475s FAIL (2025-06-26 06:18:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.081408029s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 5.788158576s FAIL (2025-06-26 06:18:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.788185527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.94225235s FAIL (2025-06-26 06:18:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.94227315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 5.745168973s FAIL (2025-06-26 06:17:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.745182388s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XX..............X......................X.] duration: 5.84897942s FAIL (2025-06-26 06:17:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.848992154s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.695775521s FAIL (2025-06-26 06:17:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.695822493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX..............X......................X] duration: 30.000606811s FAIL (2025-06-26 06:17:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000628534s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 5.545171365s FAIL (2025-06-26 06:16:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.545199248s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.169280396s FAIL (2025-06-26 06:16:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.16930318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.825117266s FAIL (2025-06-26 06:16:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.825133546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XX..............X......................] duration: 6.303479931s FAIL (2025-06-26 06:15:37 UTC) orbiter warning: context deadline exceeded Duration: 6.303501835s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 5.520202209s FAIL (2025-06-26 06:15:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.520227387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.697683026s FAIL (2025-06-26 06:15:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.697719336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.672491604s FAIL (2025-06-26 06:14:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.672516904s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XX..............X.....................] duration: 5.974792366s FAIL (2025-06-26 06:14:28 UTC) orbiter warning: context deadline exceeded Duration: 5.974814438s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.905951101s FAIL (2025-06-26 06:14:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.905976379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 5.577206849s FAIL (2025-06-26 06:14:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.577232539s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.729425247s FAIL (2025-06-26 06:13:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.729440538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XX..............X....................] duration: 6.18131623s FAIL (2025-06-26 06:13:28 UTC) orbiter warning: context deadline exceeded Duration: 6.181333595s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 5.802430652s FAIL (2025-06-26 06:13:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.802458377s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.732219818s FAIL (2025-06-26 06:13:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.732240477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XX..............X...................] duration: 6.156067108s FAIL (2025-06-26 06:13:12 UTC) orbiter warning: context deadline exceeded Duration: 6.156090434s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.652255841s FAIL (2025-06-26 06:12:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.652285439s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.774865122s FAIL (2025-06-26 06:12:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774893718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XX..............X..................] duration: 5.988207235s FAIL (2025-06-26 06:12:14 UTC) orbiter warning: context deadline exceeded Duration: 5.988227644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 5.534791239s FAIL (2025-06-26 06:11:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.534810095s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XX..............X.................] duration: 5.859683158s FAIL (2025-06-26 06:11:55 UTC) orbiter warning: context deadline exceeded Duration: 5.859731882s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.510795116s FAIL (2025-06-26 06:11:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.510816257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 5.616079512s FAIL (2025-06-26 06:11:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.616106604s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.871649522s FAIL (2025-06-26 06:11:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.871687596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX..............X................] duration: 6.090393989s FAIL (2025-06-26 06:10:43 UTC) orbiter warning: context deadline exceeded Duration: 6.090416351s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 5.532237391s FAIL (2025-06-26 06:10:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.532266337s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.859081852s FAIL (2025-06-26 06:10:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.859120287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX..............X...............] duration: 6.259714702s FAIL (2025-06-26 06:10:08 UTC) orbiter warning: context deadline exceeded Duration: 6.259731073s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX..............X..............] duration: 6.053499155s FAIL (2025-06-26 06:09:43 UTC) orbiter warning: context deadline exceeded Duration: 6.053526216s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 5.678303408s FAIL (2025-06-26 06:09:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.678333607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.861651508s FAIL (2025-06-26 06:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86166861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.22865529s FAIL (2025-06-26 06:08:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 30.228677523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX..............X.............] duration: 6.003232849s FAIL (2025-06-26 06:08:28 UTC) orbiter warning: context deadline exceeded Duration: 6.003292965s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XX..............X............] duration: 20.32460135s FAIL (2025-06-26 06:08:01 UTC) orbiter warning: context deadline exceeded Duration: 20.324616138s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)