Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 5.520202209s FAIL (2025-06-26 06:15:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.520227387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.697683026s FAIL (2025-06-26 06:15:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.697719336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.672491604s FAIL (2025-06-26 06:14:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.672516904s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XX..............X.....................] duration: 5.974792366s FAIL (2025-06-26 06:14:28 UTC) orbiter warning: context deadline exceeded Duration: 5.974814438s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.905951101s FAIL (2025-06-26 06:14:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.905976379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 5.577206849s FAIL (2025-06-26 06:14:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.577232539s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.729425247s FAIL (2025-06-26 06:13:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.729440538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XX..............X....................] duration: 6.18131623s FAIL (2025-06-26 06:13:28 UTC) orbiter warning: context deadline exceeded Duration: 6.181333595s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 5.802430652s FAIL (2025-06-26 06:13:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.802458377s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.732219818s FAIL (2025-06-26 06:13:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.732240477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XX..............X...................] duration: 6.156067108s FAIL (2025-06-26 06:13:12 UTC) orbiter warning: context deadline exceeded Duration: 6.156090434s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.652255841s FAIL (2025-06-26 06:12:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.652285439s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.774865122s FAIL (2025-06-26 06:12:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774893718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XX..............X..................] duration: 5.988207235s FAIL (2025-06-26 06:12:14 UTC) orbiter warning: context deadline exceeded Duration: 5.988227644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 5.534791239s FAIL (2025-06-26 06:11:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.534810095s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XX..............X.................] duration: 5.859683158s FAIL (2025-06-26 06:11:55 UTC) orbiter warning: context deadline exceeded Duration: 5.859731882s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.510795116s FAIL (2025-06-26 06:11:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.510816257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 5.616079512s FAIL (2025-06-26 06:11:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.616106604s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.871649522s FAIL (2025-06-26 06:11:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.871687596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX..............X................] duration: 6.090393989s FAIL (2025-06-26 06:10:43 UTC) orbiter warning: context deadline exceeded Duration: 6.090416351s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 5.532237391s FAIL (2025-06-26 06:10:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.532266337s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.859081852s FAIL (2025-06-26 06:10:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.859120287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX..............X...............] duration: 6.259714702s FAIL (2025-06-26 06:10:08 UTC) orbiter warning: context deadline exceeded Duration: 6.259731073s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX..............X..............] duration: 6.053499155s FAIL (2025-06-26 06:09:43 UTC) orbiter warning: context deadline exceeded Duration: 6.053526216s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 5.678303408s FAIL (2025-06-26 06:09:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 5.678333607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.861651508s FAIL (2025-06-26 06:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86166861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.22865529s FAIL (2025-06-26 06:08:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== equestItem_ObjectBegin Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment... Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment. Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece... Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments... Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil> Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments... Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration:* 30.228677523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX..............X.............] duration: 6.003232849s FAIL (2025-06-26 06:08:28 UTC) orbiter warning: context deadline exceeded Duration: 6.003292965s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XX..............X............] duration: 20.32460135s FAIL (2025-06-26 06:08:01 UTC) orbiter warning: context deadline exceeded Duration: 20.324616138s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.791482817s FAIL (2025-06-26 06:07:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.791502768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.230471748s FAIL (2025-06-26 06:07:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== d finished. err: <nil> Jun 26 06:07:25.478776206 (68h57m0.249417531s) [upload=55039]: Starting upload Jun 26 06:07:25.478829350 (68h57m0.249470685s) [upload=55039]: Done starting upload Jun 26 06:07:25.478906038 (68h57m0.249547363s) [upload=55039]: Uploading segments... Jun 26 06:07:25.478989260 (68h57m0.249630595s) [upload=55039]: Got next segment. Inline: false Jun 26 06:07:25.479024688 (68h57m0.249666023s) [upload=55039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:07:25.479029537 (68h57m0.249670863s) [upload=55039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:07:25.922874279 (68h57m0.693515614s) [seg_pos={0 0}, upload=55039]: Begin upload segment... Jun 26 06:07:25.922913384 (68h57m0.693554719s) [seg_pos={0 0}, upload=55039]: Done begin upload segment. Jun 26 06:07:25.922938013 (68h57m0.693579348s) [noise=true , address=104.36.82.11:4113 , piece_id=R5SOI2EWIBQ5XH2XKTRUDCD6O7ABHKLDWZL3TWQ6VUTKFXX3EKLA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.923010833 (68h57m0.693652168s) [noise=true , address=45.91.116.11:4118 , piece_id=L7W66ZTGQRHZ67YWQKDZWALRUTTWUDJPTQX2HHXEROUOECDXGCSQ, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.923306075 (68h57m0.6939474s) [noise=true , address=104.36.82.8:4969 , piece_id=434MNFELIPK7TU6BWBYN6YKSHTJF7H7XFK3DXZZX5HQUJKSOUPPA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.923405999 (68h57m0.694047334s) [noise=true , address=104.36.82.2:4729 , piece_id=CMMTF54OV2KAYEU7TLLU3PSGBR74WA5XH3Y5NARCGY2BH77527JQ, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.923551421 (68h57m0.694192756s) [noise=true , address=104.36.82.10:4079 , piece_id=3KMOTTSKOAXF7LT72VP5FFEWCTRTN77KQSNXYULOCOZRY5SAQ5SA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.923693856 (68h57m0.694335181s) [noise=true , address=45.91.116.7:4011 , piece_id=523DHPSYENCKTYDWB7QH4AKAWRFQZSIZG4HN6KBS5EO2KDYITCTQ, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.923879246 (68h57m0.694520581s) [noise=true , address=77.90.31.10:4517 , piece_id=5FD4MI2KQ6GNSIT4HE3Y2XSB5LFIZRBYLRWUEQS2NYBJH4PLDEIA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.923955202 (68h57m0.694596527s) [noise=true , address=45.91.116.13:4155 , piece_id=7EUYN7TOB3CUPSX4Y7ALEICE45GTTUI6DN4IIM3MIGCKOJE3PZBA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924117166 (68h57m0.694758501s) [noise=true , address=45.91.116.5:5298 , piece_id=HYOSAYQJQRZZNQNYOFJ42JHZN2JELMFQJ4JNHN4ZIQ6ZSGBDZ2CA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924214154 (68h57m0.694855479s) [noise=true , address=104.36.82.12:4151 , piece_id=WXNHFGC7CCFKXX6VWLJRBQCVSF77OHQBWPMPVJWNODXBDG3LBOHA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924340880 (68h57m0.694982215s) [noise=true , address=77.90.31.13:4626 , piece_id=J7Y3F3YWJBXDE5CBR6RW7N4MAZRJPPN4JQFJUA46D3IQWB7MCWUA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924422277 (68h57m0.695063602s) [noise=true , address=104.36.82.13:4186 , piece_id=I2UAYBRYPDJM7ZOWMWVBVH2SBWNID2MA3XHP6WQQP6FFPE3I7SOQ, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924547420 (68h57m0.695188755s) [noise=true , address=45.91.116.12:5226 , piece_id=DPI3RHQYXMBI36XJKDNUGC2GTX7BSTY2ZJ7CVD3UNJRA6YCKTQMA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924641101 (68h57m0.695282436s) [noise=true , address=77.90.31.18:4807 , piece_id=6VKNLGJAMJ63CWZWL3NZBC5CERCCTXWP4NPEMILSUHIZKM3JNMLA, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924740113 (68h57m0.695381438s) [noise=true , address=104.36.82.3:4767 , piece_id=NQRIQMFJE44XUEKFD4OYB3HMJC5FYAUODWBX6EDLNQTKWG5WKGGQ, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:25.924932936 (68h57m0.695574271s) [noise=true , address=104.36.82.15:4257 , piece_id=4TZJFWDGARVVEBA5DOBWJNEDMYBE5JVL4PRLCFY2FBKZ6EV3SNZQ, seg_pos={0 0}, upload=55039]: Uploading piece... Jun 26 06:07:27.582047912 (68h57m2.352689237s) [upload=55039]: Next returned nil segment Jun 26 06:07:27.582065796 (68h57m2.352707131s) [upload=55039]: Waiting for error group managing segments... Jun 26 06:07:27.874940425 (68h57m2.64558176s) [noise=true , address=104.36.82.8:4969 , piece_id=434MNFELIPK7TU6BWBYN6YKSHTJF7H7XFK3DXZZX5HQUJKSOUPPA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:27.875179148 (68h57m2.645820483s) [noise=true , address=104.36.82.3:4767 , piece_id=NQRIQMFJE44XUEKFD4OYB3HMJC5FYAUODWBX6EDLNQTKWG5WKGGQ, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:27.885616902 (68h57m2.656258238s) [noise=true , address=104.36.82.10:4079 , piece_id=3KMOTTSKOAXF7LT72VP5FFEWCTRTN77KQSNXYULOCOZRY5SAQ5SA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:27.892941819 (68h57m2.663583145s) [noise=true , address=104.36.82.2:4729 , piece_id=CMMTF54OV2KAYEU7TLLU3PSGBR74WA5XH3Y5NARCGY2BH77527JQ, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:27.902706130 (68h57m2.673347465s) [noise=true , address=104.36.82.11:4113 , piece_id=R5SOI2EWIBQ5XH2XKTRUDCD6O7ABHKLDWZL3TWQ6VUTKFXX3EKLA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:27.913537769 (68h57m2.684179104s) [noise=true , address=104.36.82.15:4257 , piece_id=4TZJFWDGARVVEBA5DOBWJNEDMYBE5JVL4PRLCFY2FBKZ6EV3SNZQ, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:27.918829838 (68h57m2.689471164s) [noise=true , address=104.36.82.12:4151 , piece_id=WXNHFGC7CCFKXX6VWLJRBQCVSF77OHQBWPMPVJWNODXBDG3LBOHA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:27.932983922 (68h57m2.703625247s) [noise=true , address=104.36.82.13:4186 , piece_id=I2UAYBRYPDJM7ZOWMWVBVH2SBWNID2MA3XHP6WQQP6FFPE3I7SOQ, seg_pos={0 0}, upload=55039]: Done uploading piece. err: <nil> Jun 26 06:07:54.837685206 (68h57m29.608326541s) [noise=true , address=45.91.116.13:4155 , piece_id=7EUYN7TOB3CUPSX4Y7ALEICE45GTTUI6DN4IIM3MIGCKOJE3PZBA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: rpc: dial tcp 45.91.116.13:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837701628 (68h57m29.608342963s) [noise=true , address=45.91.116.7:4011 , piece_id=523DHPSYENCKTYDWB7QH4AKAWRFQZSIZG4HN6KBS5EO2KDYITCTQ, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837729552 (68h57m29.608370887s) [noise=true , address=45.91.116.5:5298 , piece_id=HYOSAYQJQRZZNQNYOFJ42JHZN2JELMFQJ4JNHN4ZIQ6ZSGBDZ2CA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: rpc: dial tcp 45.91.116.5:5298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837735824 (68h57m29.608377159s) [noise=true , address=77.90.31.10:4517 , piece_id=5FD4MI2KQ6GNSIT4HE3Y2XSB5LFIZRBYLRWUEQS2NYBJH4PLDEIA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837759771 (68h57m29.608401106s) [noise=true , address=45.91.116.11:4118 , piece_id=L7W66ZTGQRHZ67YWQKDZWALRUTTWUDJPTQX2HHXEROUOECDXGCSQ, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:45726->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837769960 (68h57m29.608411285s) [noise=true , address=77.90.31.13:4626 , piece_id=J7Y3F3YWJBXDE5CBR6RW7N4MAZRJPPN4JQFJUA46D3IQWB7MCWUA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837787334 (68h57m29.608428669s) [noise=true , address=45.91.116.12:5226 , piece_id=DPI3RHQYXMBI36XJKDNUGC2GTX7BSTY2ZJ7CVD3UNJRA6YCKTQMA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:5226): protocol: rpc: dial tcp 45.91.116.12:5226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837829476 (68h57m29.608470811s) [noise=true , address=77.90.31.18:4807 , piece_id=6VKNLGJAMJ63CWZWL3NZBC5CERCCTXWP4NPEMILSUHIZKM3JNMLA, seg_pos={0 0}, upload=55039]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:36246->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:07:54.837859153 (68h57m29.608500478s) [seg_pos={0 0}, upload=55039]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jun 26 06:07:55.066518512 (68h57m29.837159847s) [upload=55039]: Done uploading segments... Jun 26 06:07:55.066542528 (68h57m29.837183863s) [upload=55039]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.230487358s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XX..............X...........] duration: 12.052176367s FAIL (2025-06-26 06:06:59 UTC) orbiter warning: context deadline exceeded Duration: 12.052196567s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.80882751s FAIL (2025-06-26 06:06:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.808850393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XX..............X..........] duration: 6.158253655s FAIL (2025-06-26 06:06:03 UTC) orbiter warning: context deadline exceeded Duration: 6.158275327s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.014721122s FAIL (2025-06-26 06:05:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.014757022s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.928932084s FAIL (2025-06-26 06:05:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.928951722s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XX..............X.........] duration: 5.965665468s FAIL (2025-06-26 06:05:04 UTC) orbiter warning: context deadline exceeded Duration: 5.965685928s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XX..............X........] duration: 6.070078893s FAIL (2025-06-26 06:04:21 UTC) orbiter warning: context deadline exceeded Duration: 6.070094995s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.875136558s FAIL (2025-06-26 06:03:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.87515875s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................XX..............X.......] duration: 5.870825296s FAIL (2025-06-26 06:03:13 UTC) orbiter warning: context deadline exceeded Duration: 5.870848641s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XX..............X......] duration: 5.998993275s FAIL (2025-06-26 06:02:56 UTC) orbiter warning: context deadline exceeded Duration: 5.999016559s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.79963147s FAIL (2025-06-26 06:02:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.799658262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XX..............X.....] duration: 5.925219776s FAIL (2025-06-26 06:02:40 UTC) orbiter warning: context deadline exceeded Duration: 5.925263401s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.718938436s FAIL (2025-06-26 06:02:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.718973275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XX..............X....] duration: 5.911135879s FAIL (2025-06-26 06:01:57 UTC) orbiter warning: context deadline exceeded Duration: 5.911146138s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.01986125s FAIL (2025-06-26 06:01:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.019886168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................XX..............X...] duration: 5.930246242s FAIL (2025-06-26 06:00:52 UTC) orbiter warning: context deadline exceeded Duration: 5.930271461s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XX..............X..] duration: 6.045391939s FAIL (2025-06-26 06:00:36 UTC) orbiter warning: context deadline exceeded Duration: 6.045441997s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.979253155s FAIL (2025-06-26 06:00:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.979277804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XX..............X.] duration: 5.924467609s FAIL (2025-06-26 05:59:53 UTC) orbiter warning: context deadline exceeded Duration: 5.924478681s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)