Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.170311886s FAIL (2025-06-26 06:51:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.170339269s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.9965031s FAIL (2025-06-26 06:50:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.996524812s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 5.734707035s FAIL (2025-06-26 06:50:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.734729549s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.250452301s FAIL (2025-06-26 06:49:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.250479493s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 5.633409817s FAIL (2025-06-26 06:49:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.633443511s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.184541079s FAIL (2025-06-26 06:49:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.184902739s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 5.69472496s FAIL (2025-06-26 06:48:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.694754938s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.143782895s FAIL (2025-06-26 06:48:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.143808435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 5.781382043s FAIL (2025-06-26 06:47:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.781404687s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.11998763s FAIL (2025-06-26 06:47:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.12000811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 5.656878711s FAIL (2025-06-26 06:46:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.656897447s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.066688051s FAIL (2025-06-26 06:46:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.066705075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 5.667499801s FAIL (2025-06-26 06:46:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.667522906s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.210053766s FAIL (2025-06-26 06:45:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.210075467s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.920595857s FAIL (2025-06-26 06:45:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.920616427s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 5.830116097s FAIL (2025-06-26 06:45:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.830135575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 5.614047138s FAIL (2025-06-26 06:44:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.614066034s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.368577602s FAIL (2025-06-26 06:44:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.368593332s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.674551727s FAIL (2025-06-26 06:44:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.674576796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 5.659928595s FAIL (2025-06-26 06:43:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.65995648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 5.782334526s FAIL (2025-06-26 06:43:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.782361769s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.974143247s FAIL (2025-06-26 06:43:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.974167463s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.716114325s FAIL (2025-06-26 06:43:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.716131768s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 51.018787393s FAIL (2025-06-26 06:42:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.01880647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 9.202976562s FAIL (2025-06-26 06:42:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.203001341s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 5.651854892s FAIL (2025-06-26 06:42:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.651874479s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......................X....................................] duration: 6.146089313s FAIL (2025-06-26 06:42:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.14610409s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.738107188s FAIL (2025-06-26 06:41:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.738130764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 5.741115755s FAIL (2025-06-26 06:41:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.741147416s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......................X...................................] duration: 6.031028853s FAIL (2025-06-26 06:41:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.031051718s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 5.671818183s FAIL (2025-06-26 06:41:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.671843152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.964352322s FAIL (2025-06-26 06:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.964370467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X......................X..................................] duration: 10.120796148s FAIL (2025-06-26 06:40:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.120812609s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.993356618s FAIL (2025-06-26 06:40:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.993376817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 5.587936856s FAIL (2025-06-26 06:40:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.587962285s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.774914665s FAIL (2025-06-26 06:40:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774940285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 5.67236866s FAIL (2025-06-26 06:39:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.672397296s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......................X.................................] duration: 6.278161676s FAIL (2025-06-26 06:39:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.278186974s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......................X................................] duration: 6.013962726s FAIL (2025-06-26 06:39:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.013978175s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.761142924s FAIL (2025-06-26 06:39:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.761185295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......................X...............................] duration: 5.917937428s FAIL (2025-06-26 06:39:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.917958057s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.928618877s FAIL (2025-06-26 06:38:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.928635008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 5.682164402s FAIL (2025-06-26 06:38:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.682186054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.82333326s FAIL (2025-06-26 06:38:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.823354442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......................X..............................] duration: 5.925079954s FAIL (2025-06-26 06:38:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.925102508s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 5.705116231s FAIL (2025-06-26 06:38:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.705141438s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.794202812s FAIL (2025-06-26 06:37:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.794234633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......................X.............................] duration: 6.254357683s FAIL (2025-06-26 06:37:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.254376328s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.722955136s FAIL (2025-06-26 06:37:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.722974241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 5.649238621s FAIL (2025-06-26 06:37:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.649262265s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)